#data-quality

#data-quality

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets. In short, it is a Spark library built by Amazon for expressing and evaluating data quality checks at scale. Besides "regular" checks and verifications, it ships some interesting features such as profiling, analyzers, and automatic suggestions, which will be demonstrated later in this post.The main library is written in Scala, although a Python wrapper (PyDeequ) is also available.To keep the focus on a single implementation, the examples in this post are written in Scala.

This engine takes topic data schemas, metadata, and test rules as inputs to create a set of FlinkSQL-based test definitions. A Flink job then executes these tests, consuming messages from production Kafka topics and forwarding any errors to Grab's observability platform. FlinkSQL was selected because its ability to represent stream data as dynamic tables allowed the team to automatically generate data filters for rules that could be efficiently implemented.

Marketers instinctively know that better data and higher-quality media drive better outcomes. But bad habits are sticky and they die hard, said Jamie Barnard , CEO of Compliant, a startup that tracks data quality standards across digital media. The ad industry has "a volume-based mentality," said Barnard, who knows that struggle firsthand. He spent nearly 16 years at Unilever as its general counsel focused on global marketing, media and ecommerce before leaving in 2022.

As technology continues to advance and companies look to remain competitive in meeting market demand, the skills that employees will need are also evolving. A growing number of companies are exploring how to address these skills and workforce gaps with artificial intelligence. HR can use AI to reveal "patterns and gaps" and benchmark "current workforce skills against evolving business needs or industry trends," said Lauren Winans, CEO and principal human resources consultant at Next Level Benefits.

Yesterday X started rolling out a new About This Account feature, which included what country the account was created from and what country the account is "based" in (which is different from "connected via"). Head of product at X, Nikita Bier, was quick to say that there were " a few rough edges," but promised they'd be resolved by Tuesday.

Within five years, saying your platform offers personalized recommendations will sound as dated as asking someone to rewind the tape. Personalization has already shifted from competitive differentiator to baseline expectation - and the transition is moving faster than most marketing teams realize. The evidence is already visible: 61% of consumers will abandon brands that miss the mark, and 65% expect companies to understand their needs without being told.

Dr. Lora Aroyo, Senior Research Scientist at Google DeepMind, argues that this assumption no longer holds up. Her research at the intersection of data-centric AI and pluralistic alignment challenges the binary worldview that underpins most AI systems. Instead of seeking a single "gold standard" answer, she advocates for embracing disagreement, diversity, and pluralism as the foundation of more reliable, culturally aware AI.

Our industry is rushing headlong toward an AI-powered future. The promise is captivating: intelligent systems that can predict market shifts, personalize customer experiences and drive unprecedented growth. Yet in that race, many organizations are short-changing or even skipping a critical first step. They are building sophisticated engines but trying to run them on unrefined fuel. The result is a quiet crisis of confidence, where powerful technology underwhelms because the marketers don't trust the data it relies on.

On the surface, it seems obvious that training an LLM with "high quality" data will lead to better performance than feeding it any old "low quality" junk you can find. Now, a group of researchers is attempting to quantify just how much this kind of low quality data can cause an LLM to experience effects akin to human "brain rot."

Your sales rep finally reaches out to a "hot lead" that marketing flagged weeks ago, only to discover that the contact no longer works there. Or worse, the company has merged, and your CRM still lists them under an outdated domain. It's frustrating, time-wasting, and surprisingly common. CRM systems are supposed to be the single source of truth for customer relationships. Yet, for many businesses, that "truth" becomes outdated faster than they realize.

Over 40 minutes, the panel returned again and again to three themes: data quality, organizational alignment and cultural readiness. The consensus was clear: AI doesn't create order from chaos. If organizations don't evolve their culture and their standards, AI will accelerate dysfunction, not fix it. Clean data isn't optional anymore Allen set the tone from the executive perspective. He argued that enterprises must build alignment on high-quality, structured and standardized data within teams and across workflows, applications and departments.

Retail media promises billions in new revenue, but for grocers, the real test is whether their data can deliver. By 2025, RMN revenue is projected to hit $176.9 billion globally, overtaking combined TV and streaming revenues and accounting for 15.9% of total ad spend ( GroupM, This Year Next Year 2024 ). For grocery retailers running on razor-thin margins, this feels like salvation.

"What we had noticed was there was an underlying problem with our data," Ahuja said. When her team investigated what had happened, they found that Salesforce had published contradictory "knowledge articles" on its website."It wasn't actually the agent. It was the agent that helped us identify a problem that always existed," Ahuja said. "We turned it into an auditor agent that actually checked our content across our public site for anomalies. Once we'd cleaned up our underlying data, we pointed it back out, and it's been functional."

The Government Accountability Office (GAO) said that data it reviewed from 23 key US government agencies (out of 24, as the Pentagon was excluded from this report) indicated there were at least 63,934 full-time federal cybersecurity employees, costing the government around $9.3 billion per year. An additional 4,151 contractors were reported to the GAO, and those cost taxpayers an additional $5.2 billion.

We were able to confirm just 11 reported incidents, either directly with schools or through media reports. In 161 cases, schools or districts attested that no incident took place or couldn't confirm one. In at least four cases, we found, something did happen, but it didn't meet the government's parameters for a shooting. About a quarter of schools didn't respond to our inquiries.

Email marketers face numerous challenges in 2025, including low engagement rates, data quality issues, accurately measuring ROI, and personalization. Experts highlight the need for collaboration across departments to overcome these obstacles.

Government workers have important jobs that are critical to providing important services to taxpayers. If jobs are cut and those services aren't provided or aren't provided in a timely and competent way, there can be significant negative fallout.

The year two status report stresses how the situation is worsening. Declining budgets, staffing constraints, and inadequate statistical integrity protections identified in the 2024 report have intensified in recent months.

What's become exceedingly important is the ability to attract and retain the best cognitive experts... to take these large models and make them very customized towards solving enterprise AI problems.

There is a complete reset in how data is managed and flows around the enterprise. If people want to seize the AI imperative, they have to redo their data platforms in a very big way. And this is where I believe you're seeing all these data acquisitions, because this is the foundation to have a sound AI strategy.

Traditional monitoring fails to meet the needs of complex data organizations; instead, engineers must develop interactive observability frameworks to quickly identify anomalies.

The evaluation tasks within the biomedical text mining community face significant limitations, particularly regarding data representativeness, quality, and participant innovation in solution development.

In our research, we successfully designed a functional material KG by employing fine-tuned large language models (LLMs), assuring traceability throughout the information process.