You do not have to choose between a secure home and a private life. By implementing "Privacy by Design," you can mitigate most risks.
Legally, you can generally film anything visible from your property that is in plain view on a public street or sidewalk. However, you cannot film areas where a person has a "reasonable expectation of privacy." This includes:
Furthermore, the rise of facial recognition in home cameras will cause the next major legal battle. Currently, laws in cities like San Francisco and Portland restrict government use of facial recognition, but private home use is mostly unregulated. It is likely that within five years, using a camera to scan every pedestrian's face and cross-reference it with a watchlist will be heavily regulated or banned.