Autonomy on the Farm

In autonomous agriculture, perception refers to a machine’s ability to sense what’s around it and react accordingly. Perception is fundamental for robotic tractors and other automated farm systems to function. Now, with the help of a select group of partners, NovAtel is testing a perception system that could enable fully unmanned farming.

“Our primary focus has been on developing a vision-based perception system that can be used in a range of cases where the dynamics outside of the vehicle are very unpredictable,” said James Szabo, agriculture autonomy product manager at Hexagon | NovAtel. “The project centres on offroad autonomy, where machines are exposed to a lot of random visual noise and where extra layers of intelligence are needed to decide whether what is being perceived poses a risk to the mission success or safety.”

The new concept, Szabo said, is built around an electronic control unit (ECU) driven by multiple perception algorithms with a combination of-mono-view, stereo and thermal cameras providing 360-degree coverage around the vehicle.

“Think about a car on a highway,” Szabo said, “where an object is going to be on a big piece of black tarmac in front of you and you want to stop before hitting it. With off-road applications, for example, in construction and mining, you’re going to have people walking around and machines moving on all sides. It’s the same in agriculture, where you’re also going to have weeds sticking up, crops and many other objects that require detection and classification. This is where you need the expertise for the system to separate the noise from high-valued objects, like identifying crops, people and even livestock.”

“Perception technologies need to be configured in such a way that balances safety, functionality, recognition and avoidance of high-valued crops, and to only focus on the objects affecting your task success,” Szabo added.

NovAtel’s expertise in this area includes powerful GNSS receivers for static and dynamic heading, and GNSS+INS systems for attitude and positioning awareness in both open-sky and obstructed environments. The company’s multi-GNSS constellation SMART7 receiver, with its built-in precision antenna, delivers industry-leading accuracy, availability and reliability, providing resilient and continuous positioning and navigation through GNSS outages. Drawing on multiple constellations like GPS, GLONASS, BeiDou, Galileo and QZSS, the SMART7 is also enhanced with deeply coupled SPAN GNSS and inertial technology.

Taking it all in

SwarmFarm Robotics, an Australia-based company, is set to begin testing NovAtel’s new perception system in the coming months.

“A wide field of view is really important in agriculture,” SwarmFarm’s CEO Andrew Bate said. “Being able to handle vibration, resonance and movement, which there’s a lot of in agricultural machinery, is quite important as well. All that movement is hard on components, but it can also impact the performance of inertial units, cameras and other sensors.”

The farm environment is in constant flux, Bate said, adding to the complexity of perception systems. “We have a living crop, so it’s different each time you look at it. A paddock is always changing, from when it’s not planted through to when it’s got different crops in it at different stages. A flowering canola crop doesn’t look like a cotton crop that’s ready for harvest.”

Some agricultural operations, such as working in almond orchards, create massive amounts of dust, another challenge for perception. “And then we have sunlight at dusk and dawn, with moving shadows,” Bate said. “Agriculture is a true 24-hour operation, so perception systems need to be able to work through all that, night and day.”

These and other factors make object perception and recognition extremely complex. But overriding all is safety.

“As far as detecting obstacles,” Bate said, “people are very much the number one concern. We need to watch out for them, but there are a lot of safety hazards in the fields, everything from four-wheeled motorbikes that have been left behind, [to] tree branches that have broken off in storms. Sometimes it’s forgotten picking crates, fuel tanks and air compressors. Centre pivots are another, of course, things that can be considered moving infrastructure.”

A camera-based system can be designed to use intelligence to classify what it’s looking at and then take a specific action.

“A lot of the sensors available traditionally in this space are very poor at discriminating between, say, a box, a car or a human,” said Craig Johnson, director of perception engineering at Hexagon | AutonomouStuff. “We’re trying to avoid LiDAR and RADAR— while they work well in automotive, they don’t really work well in the agriculture environment because of the added noise levels.”

But eliminating LiDAR and RADAR in favour of stereo vision and cameras means, computationally, a completely different ballgame. “What LiDAR and RADAR do is relatively simplistic,” Johnson said. “[RADAR is] the ‘is-there- something-there-or-not’ type of approach, very low on computational power, which is why you find them in automotive applications. LiDAR is somewhere in the middle: you can process a point cloud and you can build out your 3D maps.”

NovAtel’s new approach takes computing to another level. Processing all the high-rate data from the sensors is an extremely complicated task, requiring very advanced software stacks. This is an area where NovAtel, together with Hexagon | AutonomouStuff, will provide essential expertise to agriculture OEMs looking toward autonomous machines.

“The technology and the algorithms able to do what we’re doing have been around for 20 to 30 years,” Szabo said, “but it’s only just now that the computational power is fast enough to do it in real-time on multiple cameras, all at once. Over time, people have refined the algorithms to make the process a little bit quicker, but really it comes down to the horsepower of the CPU and the cost of the ECU. The kind of processing we need is now available, and we are at a point where the intelligence can run in real-time.”

The system can classify and track an individual object, work out a trajectory of where it’s heading and follow it across multiple cameras. “All those things are now possible at a farmer-accessible price point,” Szabo said, “and that means we’re ready for the market.”

Well, almost. Before that, there is one last thing: a huge amount of data is needed to train the system.

“This is an artificial intelligence system,” Johnson added, “and the way that works is you teach it. You present it with lots of data, lots of scenarios and situations, lots of challenges, lots of questions and answers. What is a good thing? What is a bad thing? We’re going through that iteration now with data that we’ve collected ourselves across a range of locations and scenarios, and we’re now taking the system to farmers and robotics suppliers like SwarmFarm so they can try it out, test it and give us more data and feedback from their use cases.”

NovAtel is working with customers to collect data from real settings across multiple seasons and continents to ensure the system performance meets the standards required in highly automated agriculture.

Market-led approach

NovAtel plans to integrate the new system into tier-one and tier-two agriculture OEMs. “We are an OEM provider,” Szabo said. “We supply OEM components. Our goal is to deliver a fully integrated system for a vehicle platform.”

NovAtel will educate manufacturers on what type of perception sensors to use and how to integrate them with smart software to ultimately work together in a well-honed, autonomous system. “But in the immediate right here, right now, it’s going to be the small robot manufacturers who are agile enough to be able to integrate our solution into what they have today to solve their current pain points in perception,” Szabo said.

These manufacturers don’t want to invest millions of dollars into generating an advanced perception system. But they can immediately demonstrate the value of the NovAtel system to their customers.

And that’s where SwarmFarm comes in. The company is currently using SPAN GNSS+INS technology from NovAtel for positioning, Szabo said, and they recently received perception kits to start testing the new solution.

“Our typical customers are a mixture of family farms and corporate agriculture here in Australia, and we’ve got a mix of different industries as well,” Bate said. “So, our robots are working cotton and grain farms, as well as row crops. We have robots operating on turf farms, in orchards and even in the vineyards.”

NovAtel is already helping SwarmFarm deliver huge benefits to its customers with high-precision positioning systems.

“With technologies like the SMART7,” Bate said, “we see our farmers realising savings of up to 95% on things like herbicides and pesticides, and that’s big for us because we farm in the catchment of the Great Barrier Reef, so every farmer wants to use less chemicals and of course, everyone wants that for the environment. It’s the same for the final consumers: no one wants to serve food that’s had more pesticides than necessary, so it’s all wins, for farmers, for the environment and for the consumers.”

Knowing the value and quality of NovAtel’s solutions, Bate said SwarmFarm is very much looking forward to trialling the new perception technology.

“SwarmFarm and others are going to be able to integrate our technology very quickly,” Szabo said. “It means we can get kits out there, and they’ll spend some months testing the system, doing a lot of learning in the short term.” The data will come back to NovAtel, where it will be processed and then fed back into improving the AI algorithms.

From the outset, NovAtel wanted to work with real vehicles used in agricultural operations today. “These machines have been developed to suit specific needs,” Szabo said. “They are designed the way they are, and they work the way they work because that’s the best way to do the job they are doing, the most efficient way to cover the ground. In the same way, it might be that some farmers decide it’s better to move toward multiple, smaller machines, changing the paradigm. But in reality, I think it’s going to come back to wide, fast machines.”

A range of issues

Different countries and different regions mean a diversity of topographies and cultures. “If you’re in Japan and you’ve got a one-acre paddy field,” Szabo said, “that’s going to be a particular type of machine. If we want to look at the bulk of the calories and bulk foodstuff consumed, these come from your core grains, and here you’re going to see completely different, big, heavy machines, providing bulk outputs. You have to move six to 10 tonnes of that produce out of the field per hectare, no matter what. That’s not going to change.”

“We’re not going for the easy route simply to get the machine in the field,” he said. “If we wanted to do that, we could develop a stop bar that goes on the front of the machine and call it done, but we know a physical impact sensor is not the end solution, it is limited on speed, width and versatility.”

“In the Netherlands, there might be a drainage ditch down the middle of the field,” Johnson said. “Or there could be an irrigation outlet or a power line. The localisation has the ability to overlay the geofence data and the perception data. If the two overlays don’t match, then you know that there is something different and it allows you to investigate further.”

Unfortunately, safety is still an issue on today’s farms—and many of the accidents that occur involve tractors.

“Most life changing accidents in agriculture don’t happen when operating in the field, we see collisions and crushes readying the vehicle for operation, or when servicing the machine,” Szabo said, “one unfortunately common situation is getting caught in the PTO [power take-off] between the tractor and the implement. Our perception system is designed to be able to tell the vehicle that there is a human directly behind it. From there, you can start adding additional safety features to your platform and to your machine, even if it’s not an unmanned machine. Safety of overall operations is definitely important to us and our clients, and it is a strong point of value for our system.”

Building trust

For NovAtel, one of the most difficult tasks is convincing farmers automation is not a threat to their way of life.

Around the world, most farms are still family-owned and the people who work them harbour strong emotions about what they do. There is a very direct connection with the ground they till and the crops they raise.

“The farmers we meet have a sense of pride in what they’re doing,” Szabo said, “because they know that they are essentially feeding the world. With artificial intelligence, it can look like we are taking over part of that role, where we are potentially removing that connection of the farmer with the ground.”

In reality, autonomous systems enable machines to complete tedious tasks, allowing the person normally responsible for that task to focus on other more complex duties that might bring better value to the farm and even increase productivity.

“Rather than just sitting there behind the wheel,” Szabo said, “today’s farmers and farm workers can actually be looking at the agronomics of the farm, looking at new elements and new solutions a little bit deeper.”

For Jacob van den Borne, a farmer in the Southern Netherlands, automation has already become a necessity.

“Nobody wants to work on a farm anymore,” he said. “People don’t want to do the hard work, the old traditional farm jobs, the weeding, the harvesting, so moving into automation is quite simply a must if we are going to continue.”

For a time, he said, particularly after the accession of a swath of new member states to the European Union invthe early 2000s, the Netherlands saw an incoming wave of fresh and motivated farm workers, especially from Eastern Europe. “Those days are gone now,” van den Borne said, “and it is now harder than ever for us to findvpeople to work on our farms.”

Van den Borne’s association with NovAtel began in 2012 when, after trying out a few positioning systems from other companies, he came across the SMART antennas from NovAtel.

“When we tested the NovAtel system, it was a completely different story,” he said. “That kind of availability of a clear and accurate positioning signal, even in difficult situations, like under trees in a fruit orchard, was just mind-blowing, and we immediately switched to NovAtel for all our GNSS receivers and antennas. And the thing is, the SMART7 is actually more affordable than some of the other equipment we were using.”

With the experience he’s gained working with NovAtel positioning systems and given the relationship he’s established with the company’s engineers, van den Borne said he is looking forward to seeing the new perception system in action. “I think we have come to the point when many farmers are in a position where they are ready for automation and autonomy, and I am certainly one of them,” he said.

Once farming businesses like van den Borne’s and like SwarmFarm start using NovAtel technologies, that’s when they really start to believe, Szabo said.

“We’ve been in operation for about 10 years and we use both navigation and other systems that we’ve worked in partnership with NovAtel to develop,” Bate said. “That’s ranged from GPS solutions through to operation in GPS-denied environments and carrying through to developing systems for detecting obstacles.”

“We are listening to our OEM partners to develop systems that they need and want to use to advance into autonomy. We’re not just sending them a box with a receiver and a user guide. It’s very much a partnership,” Szabo said. “So, before we send you a receiver, we’re going to sit down with you and ask about why you want that receiver, what problem are you trying to solve.”

That approach has led to an excellent working relationship, Bate said.

“Our engineers work really closely together,” he said. “We can dig really deep into the technical expertise of NovAtel, and that’s really helped us grow. We know that we can rely on their engineering and support staff to actually find solutions, and that helps us in terms of our speed to market.”

The future will one day be the past

Szabo still thinks about earlier days, he said, when the first automated and precision technologies were making their appearance in the agricultural market.

“It was right around 2004 when I was first trying to sell auto-steer to people,” he said. “It was always a very hard sell. ‘Why would I need that?’ they would say to me. Now, auto-steer sells itself. People understand the benefits. The technology has matured, but the demographic has also essentially changed. There are new people coming up who have literally grown up with the technology.”

Szabo sees the same process playing itself out in today’s autonomy space.

“It’s going to start with a few people adopting a fully autonomous machine, doing very high-value operations,” he said. “Then, the technology will become more and more integrated into traditional platforms.”

Trust will grow and farmers will become more comfortable getting out of the seat and letting the machine do its own thing, but that’s not going to happen overnight. “It starts with the first step of operator assistance,” Szabo said, “but once that uptake gains momentum, it’s going to evolve into unmanned but monitored, and then, eventually, fully unmanned.”

Looking to the future, Szabo sees big data playing an increasing role, making it possible to analyse all the factors that have an impact on farm productivity and quality of outputs in closer and closer detail. “And then the important thing is, actually, what can you do about it?” he said. “How do I actuate that data into a physical action that is going to have a tangible effect on either the cost of my inputs or the value of my outputs? I think that’s going to be the next big thing.”

It’s an interesting time, to say the least, for ag robotics.

“We’re seeing product-market fit now and commercial adoption,” Bate said. “In Australia, autonomous robots are actually becoming quite common in some regions. We’re only just starting the perception work now. This is our next big project. It’s exciting, it’s great to be a part of it and NovAtel is the kind of good partner we know we need in order to succeed.”

Read the full PDF here: