Why financial institutions are abandoning screen scraping for open banking

In the banking world, screen scraping has been around for over 25 years – it’s how many fintechs got their start. But a continued push towards open banking in Europe and the latest regulatory moves in North America – such as the U.S. CFPB’s final open banking rule – aim to do away with the practice.

Beyond the compliance aspect of governing data sharing practices, forward-thinking financial institutions are already engaged in their next evolution, moving away from screen-scraping to further integrate financial services into a broader digital ecosystem.

Let’s look at how screen-scraping works, why it can be problematic and even risky, and how open banking is the secure solution needed to move beyond screen scraping and seize new opportunities.

What is screen scraping?

Screen scraping (also known as web scraping) is what a developer might do to get access to information that’s usually only shared via a webpage. In the context of banking and finance, screen-scraping allows a third party to copy information displayed on a screen (such as spending or account balances) and using that data for another purpose, such as financial aggregation, budgeting, or loan applications.

To put it simply, web scraping is driven by bots/web crawlers. They work in the same manner as search engines do – Fetch and Copy. But web scraping focuses on retrieving particular data from a website, whereas search engines frequently scrape the majority of webpages on the internet.

In the early days, companies used web scraping to take advantage of their competition. If you wanted to keep tabs on a competitor’s product pricing, steal their leads, hijack their marketing campaigns, divert APIs, or just blatantly rip off their content and data… web scraping was the way to go.

​​Once you have access to the raw data, you can reuse it to create your own page (or pages) with your own colors, brand, structure, and so on. The possibilities are endless.

How screen scraping works

The technique of screen scraping is especially useful for data aggregation. For example, a company that wants to consolidate points and status for an individual across all the person’s airline mileage accounts could offer a point-tracking portal.

When used ethically, customers agree to share their credentials in this manner with a third party – let’s say, a fintech. The third-party app logs in using the consumer’s login credentials.

You’re likely familiar with Mint, the personal financial management website and mobile app: it has its roots in the technique, and now uses a third-party solution that employs screen scraping and other means to aggregate bank accounts, credit cards, investments, and bills all in one place.

The third party now can access the customer’s data, and the customer now has access to new financial applications. The third party is happy as they have a new customer, and the bank is blissfully unaware. All good, right?

The problem with screen scraping

This fragile balance (between customers/scrapers and data-holders/owners) is one driver of Open Banking regulations (such as PSD3 or Dodd-Frank Section 1033) which try to answer the question of who the data belongs to.

Even when it’s used ethically and with users’ consent, there are some cracks in screen scraping, which has downsides for all parties concerned: the customer, the third party, and the bank or credit union.

The impact of screen scraping on banking customers

For customers, the user experience may not be consistent.

• The third party accessing the customer’s data is at the mercy of the bank’s HTML: ultimately, screen-scraping is a hack, a workaround. If the website changes even a little, the data may not be found until the screen scraper adapts. It’s a constant battle where the customer loses.
• It’s slow because a lot of data must be downloaded and processed just to get at a few necessary bits. To stay up to date in case there are changes, that data must be downloaded frequently.
• It stops working because it’s an us-vs-them situation, and the impacted companies are working to prevent this from happening. When those companies are successful, it stops working for the customers.

Screen scraping can also pose security risks: there are no set standards, and the passwords customers share are stored in plain text, making them more vulnerable to hacker attacks.

That is a risk to the consumer and the financial institution, because credentials for accounts they own are stored on someone else’s infrastructure.

The impact of screen scraping on third party providers

Screen scraping also causes a major inconvenience for the third party every time a bank changes their login page or interface. It’s estimated that these types of transactions fail at least 30% of the time, resulting in unhappy customers and headaches for the third party.

The impact of screen scraping on financial institutions

Finally, the bank is impacted as they now need to handle the rise in calls to their backend servers.

• Screen scrapers are “hitting the website” as if they were a logged-in user. However, they are not human, so they can hit the website much more frequently. And they hit it more frequently to stay up to date.
• Also, they download a lot more information than they need (they need the whole page, including HTML/CSS, and everything present on the page, even if they just want a line item) because that’s all they have access to – pages of data (instead of specific data fields).
• Systems designed for a single human user must now handle a steep increase in non-human based calls to their systems.

See also: 6 things you need to know about screen scraping in banking and financial services

Screen scraping vs. open banking

There’s a delicate balance between eliminating screen scraping and maintaining the convenience of sharing your data with a third party. This is where open banking comes to the rescue.

Open banking offers a common, open API standard to communicate and exchange data, with all the built-in security protocols such as FAPI and OIDC, allowing for efficient and secure data exchange.

Dive deeper here: What is Open Banking and How Will it Affect You?

Between market-driven innovation and consumer demand, there’s a growing call to move away from screen scraping.

Over the past decade, the escalating demand for innovative services and seamless digital experiences has pushed banks to evolve rapidly. This transformation is not just about technology but also involves adapting to new consumer behaviors and expectations.

Nearly half (45%) of customers of US national banks profess to already use or be interested in the concept of open banking, according to a 2023 Mastercard survey.

Today, the regulatory environment governing data sharing practices has become increasingly stringent and forward-thinking.

The Dodd-Frank Act in the US, including specific sections like 1033, is shaping how data must be handled securely and transparently. In Canada, the government has taken notice of the pitfalls associated with screen scraping.

By enabling fast, secure access to quality data through APIs, these regulations aim to curb screen scraping and misuse of financial information while also reducing bias and reliance on credit scores. The next evolution will further integrate financial services into a broader digital ecosystem.

See also: Axway’s Role in Shaping North American Open Banking Standards

What this means for banks and financial institutions

The market is resolutely marching toward adoption of open financial APIs, with FDX emerging as a front-runner standard in North America.

In this new landscape, closed data silos are unable to respond to the modern speed of business. Moreover, a decentralized system built upon a common, open standard prevents the emergence of a single dominant player, one of the fundamental goals stated by the CFPB.

To evolve beyond screen-scraping, financial institutions (FIs) will need to implement two key capabilities around the APIfication of their IT systems:

• Consent management: instituting a sophisticated consent management system is crucial to maintain consumer trust and regulatory compliance. It allows FIs to participate in BaaS and Platform Banking, forming partnerships with the fintech ecosystem.
• Consent dashboarding: for better transparency and end-user control over data shared.

While there is undoubtedly a cost to compliance, such as technological upgrades and system integrations, more and more FIs recognize the core benefit of open banking: the ability to reach and capture new customers they wouldn’t have access to otherwise.

• 74% of banks see collaborative business models as crucial to succeed with their future business strategy, according to Sopra Steria’s latest Digital Banking Experience Report.
• Almost half (48%) aim to invest in improved APIs to connect more effectively with partners.
• Per Axway’s 2024 State of Enterprise API Maturity report, financial services enterprises are one of the industries that most leverage direct API monetization to build their external ecosystem.

Through Banking-as-a-Service (BaaS) and other platform banking trends, forward-thinking FIs are redefining how banking products and services are consumed.

Embracing an Open Finance ecosystem

To fully participate in this expanding financial ecosystem, an open banking solution with a federated API management component is an essential tool to help support a bank’s digital business strategy, governance, and security for API products.

Amplify Open Banking addresses the complexities of open banking regulations for financial institutions and effectively integrates their services into digital ecosystems. It ensures compliance with laws like Dodd-Frank 1033, manages consents meticulously, and supports collaborations with fintech, reducing overheads and accelerating innovation.

Beyond just providing a more secure and reliable alternative to screen scraping, Amplify Open Banking can help FIs shake up the world of finance by offering their customers more ways to save money, move money, and improve financial well-being.