It is the time of the year when I am getting ready for one of my favorite events in New York – the LDV vision summit – taking place in late May. In addition to seeing many of my friends there, I know I will be infused with the excitement, dreams and breakthroughs of the awesome scientists, entrepreneurs, and now investors who all share the same vision: computers will, one day, see the world the way humans do. This will create tectonic changes in our world. At an amazingly larger scale.
You’d have to be living under a rock to ignore all of Machine Learning and AI’s hype right now. What you might not know is that Computer Vision is an essential, vital milestone towards true machine intelligence. Aristotle defined perception as a necessary precursor to intelligence (Nous).
Perception is deeply intertwined with intelligence. So solving the perception problem for computers opens the path to real machine intelligence. We computer vision people are addressing one of the five senses of perception. As a result, even though this might freak out Elon Musk, we, as a community of technologists, are getting closer to true machine intelligence every year.
The Vision Summit and its growing popularity is good evidence of that. If you have not registered yet, I strongly encourage you to come and attend these two days of exciting and bold Computer Vision related keynotes, panels and competitions. You’ll witness how mature our field has become, and how quickly it is still progressing.
Also, don’t miss our keynote session. See how Placemeter’s computer vision algorithms help our cities and retailers get more efficient. See how we help the world get ready to handle the unprecedented 2.8 Bn people population growth that is coming in the next thirty years.
‘Eyes on the street’: 311 and the sidewalk ballet by Melissa Sands
The sidewalks of New York City were a favorite subject of writer and activist Jane Jacobs. In The Death and Life of Great American Cities (1961), she described the city as an “intricate ballet” of movement and change that plays out on the sidewalk “…bringing with it a constant succession of eyes.” Jacobs believed that “eyes upon the street”—by which she meant the watchful gaze of neighborhood shop-owners and residents, alert to minor disturbances and problems—are essential to a vibrant, safe city.
In 2003, the city’s “eyes” gained a direct link to city government with the launch of NYC 311, NYC’s non-emergency customer service line. NYC’s 311 line operates 24/7 and assists 60,000 callers per day. Residents and visitors report everything from noise complaints to broken parking meters to rodent sightings. Those who contact 311 are sentinels of the sidewalk, supplying vital information about what’s happening on the ground to the city agencies responsible for fixing problems.
Does higher foot traffic, and therefore more ‘eyes on the street’, translate into more 311 reports?
A high density of pedestrians means greater potential for complaints about a blocked sidewalk, an overgrown tree branch, or an overflowing litter basket, but what does this mean for 311? Armed with data from Placemeter, which measures activity on streets and sidewalks from a network of sensors, and NYC OpenData, I set out to study how rates of pedestrian traffic dictate what gets reported to 311.
First I combined NYC 311 reports with Placemeter-generated foot traffic data from over the course of two weeks last summer. Restricting the analysis to data-rich Midtown Manhattan, I spatially interpolated hourly daytime foot traffic for every inch of Midtown. On top of the foot traffic layer, I then overlaid NYC 311 data from NYC OpenData for the same period and extracted predicted values of hourly foot traffic for each geocoded 311 report. In other words, each 311 call is linked with how many pedestrians, on average, were in the vicinity of the problem when it was reported to the City.
The first take-away is that, indeed, more people means more ‘eyes on the street’, but only up to a point: when foot traffic is at its highest, 311 reporting tends to drop off. One possible explanation is that who is on the street matters; highly trafficked areas are often those that swarm with tourists, who would be less likely than residents to report problems to 311. Another potential explanation is the bystander effect, a social psychological phenomenon in which the presence of many other individuals decreases the chances that any one person will report a problem.
The second take-away is that the relationship between foot traffic and 311 reporting varies by the type of complaint. For certain categories of 311 complaints, the number of calls is relatively unaffected by the quantity of foot traffic. For example, in Midtown complaints about sidewalk condition are generated at a similar rate regardless of how many people are around.
Calls regarding air quality spike around 800 pedestrians per hour but quickly fall off in areas of middle to high foot traffic. Street noise, one of the city’s most commonly-reported grievances, peaks around 1,200 pedestrians per hour but is rarely reported at more bustling midtown locations. This may reflect expectations about where noise is acceptable, e.g., we expect Times Square to be loud but are less tolerant of the urban cacophony when it resonates on a residential block.
Meanwhile, reports of homeless persons in need of assistance are rarely made in the least trafficked areas but are over-represented in places with a high density of foot traffic.
Nearly a half century after Jacobs coined the term ‘eyes on the street’, the combination of Placemeter and 311 data allows us to explore the nuances of an idea that has influenced urban planners around the world. These novel data sources can breathe new life into Jacobs’ ideas about the vitality of the city and its sidewalks. They can help us to better understand how pedestrians engage with their city and its intricate sidewalk ballet.
Melissa Sands is a PhD Candidate in Government at Harvard University
New Yorkers love to hate Times Square. It’s touristy and full of cheap souvenir shops. It’s a complete assault on the senses: lights, jumbotrons, comedy club promoters, jugglers, musicians, half-naked cowboys… It’s a mess.
Still, it’s hard not to be a tad impressed.
1. It really is worth seeing all lit up in the evening:
It is. At least once.
2. It’s way safer and enjoyable for pedestrians than it was just five years ago:
Worse than a packed crowd, you would’ve had to deal with the added nuisance (and threat) of speeding taxis and buses! Thanks to the NYCDOT, the whole plaza was redesigned to better accommodate large flows of visitors with the shutdown of Broadway. Injuries fell by 35% and 80% fewer pedestrians have to walk in the roadways.
3. According to our data, there are still a few moments where you can be sure to avoid crowds:
Our suggestion: avoid weekends. Either take a day off and come on a weekday morning between 10:00 AM and 2:00 PM, or if you can come at night, pick a weekday evening after 8:00 PM.
4. You can maximize the comfort of your visit by walking Southbound instead of Northbound:
More people walk Southbound in Times Square. So go with the flow, you don’t want to be fighting your way through!
5. One of the coolest outdoor food and music festival takes place there every year
Even New Yorkers agree that you can’t miss it! Be sure to grab a bite of the finest tastes around the globe on June 1st (and its pretty cheap!).
In the end, even if some locals may dread Times Square, no one can deny it’s a special place, unique enough to attract millions of visitors each year.
We should also celebrate it as an historic milestone in improving pedestrian safety and access in New York.
N, Q, R, L, 4, 5 and 6 lines, Union Square is what one could call an “intracity hub.” Thousands of people transit here every day, at the surface or underground (92,633 per day during the last week of February according to MTA data): students, workers, tourists, shoppers, street dancers and early-day joggers, and many, many more we don’t have time to list.
Our block (14th St. between Union Square West and 5th Avenue) sits on a mixed commercial and residential district. You will find almost every restaurant chain you can think of, a cupcake bakery, beauty bars, multiple shoe and clothing stores, and a particularly high-end gourmet supermarket. But the most striking feature of our block is the undeniably impressive 375,000 square-foot New School University Center, which hosts and brings several hundred students in the neighborhood every day.
This week, we wanted to focus on the biggest happening over the last seven days or so. Namely, the end of winter. Here is a snapshot of what a few days of warmth has done to Union Square:
We went from a freezing 14°F and 7.5 inches of snow to bright sun and the most comforting 54°F ever in only four days. And this is very observable in the foot traffic data from Placemeter:
It is rather intuitive that weather would affect foot traffic, but it’s always fun and interesting to verify with hard data. Gaze at how the temperature shifts are echoed by the traffic volumes.
Taking a look at weekdays and weekends averages, it’s easy to see that Union Square really is a hot spot in New York, where people go to work, study, but mostly visit and shop. Both weekdays and weekends have a rather slow start, but big steady 11:00 AM-7:00 PM pedestrian flows.
Here, weekdays peak at noon, versus 5:00 PM on weekends, an unusual characteristic we see at only the most vibrant places. Looking at these trends, one of the retailer on this stretch could decide to increase his staff between 4:00 and 6:00 PM on weekends to cope with the influx of shoppers, and then decrease it after 7:00 PM.
The directional foot traffic data shows that between 7:00 AM and 9:00 AM on weekdays, people are going more toward Union Square, most certainly to catch their train to work. The same trend is even more visible on weekends, where people have a high tendency to walk from 5th Avenue toward Union Square from noon to 6:00 PM, showing the attractiveness of this major New York plaza. This is valuable information that could help retailers adjust and improve their strategies to convert more passersby into customers, like orienting accordingly street advertising depending on the time of day.
Placemeter generates pedestrian & vehicle counts and location intelligence by applying our proprietary cloud-based algorithms to video sensor streams and public video feeds. In addition to these counts, retailers can use foot and vehicle traffic patterns to determine if passersby are locals or tourists, commuters or leisure seekers, and add in the effects of weather, season, and local events. Our platform works with existing security cameras or simple IP cameras that can be set up in minutes. We achieve 85+% accuracy, continuously improving over time, and samples 100% of the population, not just people with an app or a phone.
Placemeter works with a variety of retailers, from large beverage chains to upscale boutiques to retail showrooms, to optimize store operations at existing locations and improve site selection for new locations.
One retailer, a boutique gourmet food chain based in New York, was already using Placemeter data to measure foot traffic and conversion rates at its existing locations. When they decided to open a new location in Manhattan, their CFO turned to Placemeter to make an informed decision.
First, the retailer conducted their own analysis of the city landscape to determine which areas their target audience frequent, compared against existing locations. They decided which areas were most desirable based on a combination of census and market data, competitive landscape, and local knowledge on the “brand” value of certain regions of NYC.
Stores were already established in 2 of the top 5 regions.
Once the retailer had narrowed down their selection down to one area—Union Square—of the final 3, based on demographics, market research, and existing locations, they decided on 4 available leases.
With the agreement of their real estate partner, the retailer set up Placemeter sensors at all 4 locations to measure walk-by traffic and calculated the dollar potential of each.
Surprisingly, while some locations were on busier intersections or more iconic streets, the storefront on a side street was ultimately chosen based on Placemeter dataon walkby traffic, due to its location on a wider, more walkable sidewalk that was more welcoming to pedestrians.
By analyzing trends by day of week and time of day, the retailer was able to conclude if walkby traffic was more likely to be strolling tourists (their target market) or commuters. They found that a pedestrian at 2:00 PM is more valuable than a pedestrian at 9:00 AM, because they are more likely to come in and buy something. The location they chose had the best combination of high volume and right volume at the right time, thus maximizing the store’s revenue potential.
They recently broke ground on their new location and are looking forward to opening their third location in NY in Spring 2015! Now, in both their existing and new locations, the retailer is able to use Placemeter data to streamline store operations, using total traffic and conversion rates to:
Test marketing efforts
Allocate store resources (staffing, facilities) dynamically
The foot traffic data we’re looking at comes from Bedford Avenue between N. 10th Street and N. 11th, from a nine-day period starting in late February.
Tree-lined and residential, this short block of Brooklyn’s longest street takes only a minute to walk end to end. In that short distance, you can find waffles on a stick,* an alehouse with a daily-updated beer menu, multiple brunch spots, and “a rustic schoolhouse-styled bar.”
* Editor’s note: while not making an official endorsement, at least one BONY team member has vouched for the organic frozen yogurt and waffles at Von Dolhens.
The first thing we noticed was very different behavior on weekdays vs. the weekend.
During the week, many commuters travel back and forth between their homes in Williamsburg and their jobs in Manhattan, visible in our data as the weekday morning and evening rush hours. A small spike around 6:00 AM indicates to us that some workers still have to go in on the weekends (or is it due to late night dancers coming back from Good Room, just North of McCarren Park in Greenpoint?).
Our block lies just a few streets north of the Bedford subway station, one stop from Manhattan on the L train. When we subdivided the data by direction, the weekday commuter pattern became even more clear with a swell of southbound morning traffic and corresponding northbound evening traffic.
Williamsburg has been residential for a long time, but the neighborhood is also in the middle (at the end?) of a demographic shift. After deteriorating in the 1960s, it began to bounce back in the 1990s as artists and musicians were squeezed from SoHo and the East Village. In recent years, gentrification has started to push the hipster crowd further into Brooklyn as more and more yuppies balk at Manhattan rent, completely transforming the neighborhood.
One result of this shift is that Williamsburg has become over the years a popular spot for weekend escapades with numerous brunch options, flea markets, and attractive parks. This behavior leads to large spikes in traffic around 2:00 PM and 6:00 PM on Saturday, although traffic drops precipitously on Sunday evening.
While Sunday evening might be expected to be quieter than Saturday, this particular Sunday had the biggest snow day in our dataset, quieting much of New York later in the day. The large downward spike around 4:00 PM turned out to be the sun hitting the dusty window in front of this particular sensor, problem solved by cleaning it.
There’s always more insight to be uncovered in this sort of dataset, and we hope that this post has gotten you thinking about questions that these data could help you answer. For example, one member of the Blocks of New York team has taken the (very) data-driven decision to come to work at 10:30 AM instead of 9:00 to avoid the crazy L train 8:00–9:00 AM morning rush — nothing to do with beauty sleep of course. On the contrary, Van Dolhens could decide to open earlier in the morning to capture more customers grabbing an organic frozen yogurt on their way to work.
This week, after our little escapade in Brooklyn, we are venturing to one of New York City’s most iconic neighborhoods. Famous as the home to your favorite TV characters from Seinfeld and Will & Grace to 30 Rock’s Liz Lemon, the Upper West Side is quintessential New York.
According to Placemeter, this is the average pedestrian traffic you’ll see on the block per hour in the winter for a weekday and the weekend (sampled from about a three-week period in January and February).
Our data shows that the weekend and weekday activity follows the same pattern but small differences are explained by…
“Chloe, zoom in to see the license plate…” / “Let’s enhance…”
It’s a classic scene in 24. Something bad happens under the scope of a lo-fi analog video cam that Chloe’s hacked into. Or a major terrorist attack takes place right under a satellite positioned above that specific area—it’s always amazing how fast Chloe can reposition satellites, but that will be the topic of a future blog post. The bad guys are getting away, and the good guys (Jack Bauer of course) can’t let them go unpunished. He needs the license plate number of that fleeing car. So Chloe zooms, zooms, and zooms more, and the license plate appears. Crystal clear. Well, this is just in the movies. At least for now.
[Update]: Thanks to Vinnie Quinn for pointing out to us this amazing collection of “enhance…” TV & movie quotes that perfectly sums up what we’re talking about!
To get a better sense of what we’re talking about here, let’s see how image sensors work, starting with the human eye. Our retina is a little flat disk in the back of our eyes that has a limited, yet very large number of light and color sensitive cells (6 to 7 million!) called cones and rods. A lens in our pupil concentrates light rays into this disk, causing cells to react and create an array of values—or a “natural” image—that is then encoded and transferred to our brain for processing or storing.
Cameras have a fixed field of view and a matrix of elements that spans that same field of view, meaning they have a built-in ceiling for accuracy. You can zoom in easily if what you see on your screen is not the full resolution image; when you get to full physical resolution, you can still go a little further and correct small noise effects or blurs, using block-toeplitz matrix deconvolution for example (the first thing I studied on the way to my computer vision PhD). This is what you can get:
But that will NOT get you a clear view of a license plate that is just a dot in the original full resolution image.
All in all, an image sensor, e.g. a camera, is only as good as its individual components: it takes a really good lens and a really huge array of CMOS or CCD to build high definition images. The largest existing one has 3.2 Gigapixels, but more common ones are around 60 Megapixels.
Of course, closer is better, but lower satellites have to fly faster to stay in orbit, so they are not really useable for static surveillance.
The finest satellites today have a resolution of 30 cm (1 pixel=30 cm). Now, a license plate in the US is 12”x6”, or 30.5×15.2 cm, which is 1×0.5 pixels at 30cm resolution. This is what it looks like:
Not quite at the level we need to see license plates, but close—even if it does not look like it.
As camera technology progresses, we’ll eventually see low-orbit satellites that have the proper resolution to do such things, but it’s some years down the road.
An interesting development that could soon make its way into satellite technology—and improve their resolution—is the concept of “light field cameras.” At its most basic level, a light field camera contains a bunch of microlenses that act like one huge lens. Images taken from light field cameras capture a much larger field of view, with much greater detail in an image, significantly improving resolution along the way.
One of the effects of light field camera technology is the ability to focus in on every different part of an image with much greater clarity than standard cameras. Another is the ability to 3D model objects captured in the 2D dimension of an image. True 24-level stuff.
You can already buy a consumer version of this in the Lytro camera. Who knows when light field cameras will be introduced to satellite technology. One things for certain: it’ll definitely be too late for Jack Bauer.
By the way, we’re still looking for a Computer Vision Engineer and a Computer Vision Summer Intern. Apply here.
Unlike the major commuting corridor or the hip shopping haven that we’ve covered in the past, this block doesn’t present any singularly-attributable foot traffic pattern. While this may sometimes point to a lack of unique character, in the case of Livingston between Smith & Hoyt, it is entirely because of its unique character.
While we see some traits associated with office-dense areas, like local peaks during lunchtime and evening rush hour, other observations seem counterintuitive. Sustained volume on weekdays suggests that most of the traffic can be attributed to shoppers or people not confined by typical work hours (Empire State University is on this block), and late evening activity is a sign that people live nearby.
But how can there be a strong evening commute with no morning commute to speak of? Our theory is that this block is neither the origin nor the destination for much of its commuter traffic, and people tend to take different routes in opposite directions. This paper from Northwestern University offers an interesting discussion of this phenomenon.
Three weeks of daily traffic volumes broken down by direction of travel confirms our hypothesis: between January 28 and February 17, westbound traffic was consistently higher than eastbound, by an average of 20%.
Macy’s entrance on Livingston accounts for as much as 37% of the daily foot traffic on the north side of the block on average. This kind of data gathered on an extended period of time could help Macy’s get a real sense of its power of attraction in the neighborhood over time and allow them to accurately measure the effectiveness of different strategies (storefronts designs, street marketing, etc.) to capture more foot traffic on days that are relatively low in terms of sidewalk-to-store conversion.
Our last two posts used Placemeter data to cover blocks in Midtown neighborhoods with a strong mix of retail and business addresses. We showed how the pedestrian movement data can tell you a story of people moving to and from work, happy hour, and shopping. What happens, though, when you analyze the data from a less predictable block? See below in our third installment of Blocks of New York!
SoHo is a world famous neighborhood located in an already world famous city. Once the epicenter of NYC’s avant-garde art scene, SoHo is now the epicenter of NYC’s slightly less avant-garde, but massively more lucrative, art gallery and retail scene. Despite its rather commercial nature and sky-high property prices, SoHo somehow maintains a charm that makes it one of the few tourist-centric places that most bonafide New Yorkers would gladly visit.
Today we’re looking at one such block, cobblestones and all: Crosby Street between Grand and Broome.
Though the weekend and weekday trends are relatively similar, their differences can tell us something about the subtle nature of quiet SoHo blocks.
Notice how on the weekday, there’s a steady, steep…