Streamlining Data Integration with Views CSV Source: Insights from Daniel Cothran

Enhancing Drupal's Capabilities with Efficient Data Handling and Community-Driven Development
Streamlining Data Integration with Views CSV Source: Insights from Daniel Cothran

In web development, having the right tools can significantly streamline processes and improve outcomes. The Views CSV Source module, created by Daniel Cothran and Nia Kathoni, provides a straightforward yet powerful solution for Drupal users looking to integrate spreadsheet data into their websites. This module allows developers to use CSV files as the base table for a View, eliminating the need for additional database tables or numerous modules and simplifying data display on web pages.

Nia Kathoni, a Senior Software Developer at JSI Research & Training Institute, Inc., contributed his extensive industry experience to the project. Daniel Cothran, who has a public health background, combined his technology and community service expertise to help develop this tool. 

Throughout his years working with Drupal, Daniel often faced the challenge of presenting spreadsheet data on websites, whether in simple tables or complex charts. Traditional methods were usually labor-intensive and resource-heavy. The Views CSV Source module overcomes these challenges by enabling easy data manipulation directly from CSV files, such as filtering, sorting, and performing calculations.

In a conversation with Kazima Abbas, a sub-editor at The DropTimes, Daniel shared his journey into web development and the creation of Views CSV Source. He explained how this module not only simplifies the data presentation process but also improves the efficiency and performance of Drupal sites, making it especially valuable for projects requiring reliable and streamlined data handling.

When asked about his journey with Drupal, Daniel explained that his company initially hired him because of his public health background and writing skills. He was tasked with summarizing and posting resources developed by his project. At that time, the project’s Drupal 6-based website had been recently taken in-house, and he was coordinating with a consultant for any development needs. However, he wanted to handle these tasks independently, so his company provided him with a brief training session. In his spare time, Daniel used Drupal to build a website for a nonprofit organization he started with colleagues from graduate school. His supervisor, impressed by his progress, entrusted him with independently building another project's website. The success of that project attracted further interest, leading Daniel to expand his team to meet the growing demand.

"I pursued public health because I wanted to make people’s lives easier, and I feel like Drupal helps me achieve that, and in a way where I can more clearly understand my contribution. I helped jumpstart the community to apply to be a Digital Public Good because I already saw Drupal in that light and wanted others to see it that way too. It’s the way it is because of the community, of which I am happy to be a part."

He added,

"In addition, Drupal’s flexibility and robustness (especially for government-funded websites) enabled me to continue to advocate for its use within my organization."

Daniel Cothran

 As a Senior Technical Advisor and Drupal Lead at JSI, Daniel's main responsibilities include ensuring that the Drupal-based websites meet the needs and requirements of various project funders, such as government agencies, foundations, and educational institutions. He coordinates efforts across projects and shares best practices with his teams and other colleagues, ensuring that the websites serve their intended audiences effectively.

Daniel explains that he manages cross-project teams of web designers, developers, and content managers by likening his role to leading a digital agency exclusive to his company's projects. He notes that his team comprises a wide range of skills and can rely on other company personnel as needed, similar to consulting specialists. Daniel emphasizes the challenge of orchestrating these skills to meet the needs of clients and colleagues, but he credits his success to having great colleagues. He strives to ensure that his team members are fulfilled in their work and motivated to tackle new challenges.

When asked about the motivation behind developing the Views CSV Source module, Daniel explained that it was about "scratching our own itch." He and Nia Kathoni often received files with data they wanted to present on the web, but the process of transferring data from a file into a database was cumbersome. He detailed the typical workflow, which included installing multiple modules like Storage, Feeds, and others such as Tamper and Feeds Tamper; configuring a storage entity type with the necessary fields, vocabularies, and permissions; setting up a feed type, including tampers; and finally importing the feed. This process was not only labour-intensive but also led to a bloated database, often unrelated to the overall content strategy. The Views CSV Source module was designed to streamline these steps and prevent database bloat, making it a more efficient solution for handling data presentation.

Daniel outlined the following key functionalities and benefits of the Views CSV Source module:

1. Bridging the Gap Between Data and Drupal:

  • It’s wonderful when your Drupal site is the generator of data, but there are many times that it’s not. If you’re lucky, you have a remote database that you can access through a module like Views Database Connector (one of the modules I maintain, by the way). But often, the data don’t exist in a database. When they don’t, you might have to do what I describe above. Or you might need to hand-code HTML. Views CSV Source pulls data directly from a CSV file that you upload to your site or from a remote file.
  • Drupal already has robust data handling abilities through Views and contributed modules (including some of the ones I maintain like Charts, Views Simple Math Field, Views Timestamp to Date, and Views Fields On/Off). Views CSV Source brings all the features of Views to data that remains in a CSV file.
  • Views CSV Source can connect to data that requires basic authentication in order to be viewed.

2. Handling Dynamic Data:

  • I had a project before this module where I needed to pull data from a remote source (the World Health Organization) into my database, and I knew the data changed sporadically. I was able to take advantage of the scheduling feature provided by Feeds to ensure my data were updated without me needing to do anything manually. However, I was always nervous that some of the data did not update during one of the feed runs, and it was difficult to verify that due to the volume of the data. With Views CSV Source, if the source file updates, the data presented in your View will too.

3. Handling Medium-Sized Data:

  • Traditional (disaggregated) Views load the entities and because of this, they start to struggle when trying to display more than about 400 rows at a time. Views CSV Source can handle several thousands of rows of data before slowing down. 

4. Portability:

  • This is a bit niche, but as a maintainer of the Charts module, I often see support requests that are very difficult to respond to because I can’t see or manipulate problematic views. With Views CSV Source, you can export/import a view and all its data between sites without needing any other content management. This was not a benefit I anticipated, but I’m loving it.

Daniel highlighted the significant improvements in efficiency that the Views CSV Source module offers compared to traditional methods,

"Views CSV Source improves the efficiency of Drupal development teams by reducing the steps to display data and improves the efficiency of Drupal itself by reducing the number of modules needed, the number of database tables required, and the memory needed to display medium-sized datasets."

Daniel emphasized several key features of the Views CSV Source module that set it apart from other solutions. He mentioned that while the Views JSON Source module served as an early model, it could not aggregate data. To address this, Daniel and his team completely rebuilt Views CSV Source to better resemble the underlying structure of Drupal Views, enabling robust data aggregation.

Regarding performance and handling large datasets, Daniel noted that while formal testing and benchmarking are limited, the module has shown impressive capabilities. He connected it to a remote dataset containing 24,937 rows, and it rendered the data in approximately three to four seconds. This efficiency is primarily attributed to the module's integration with the thephpleague/csv library, which uses PHP's stream capabilities. This allows the module to process CSV documents with millions of records while maintaining a minimal memory footprint, ensuring that it can handle large datasets effectively without sacrificing speed or encountering memory constraints.

Daniel highlighted the integration capabilities of the Views CSV Source module with other Drupal modules and external systems. He explained that the module supports various formats provided by Views, including those from other modules like Charts. Users can add columns from the CSV as fields and utilize other modules, such as Views Simple Math Field, to interact with these columns. The module also supports filters, including contextual filters, sorts, and aggregation features for grouping, counting, summing, and averaging data. While there may be some limitations, the experience is designed to be similar to using an entity-based view.

He further elaborated, stating, 

"In terms of external integration, you can connect to a remote CSV file (or link that outputs a CSV file), and this can still work if the file requires authentication to access the contents. You could also use this module to generate your own CSV download through Views Data Export or you could generate a JSON file from your view that could be read by other systems." 

This flexibility allows the module to interact seamlessly with external data sources and systems, enhancing its utility in various applications.

Daniel explained that the Views CSV Source module remains user-friendly by using Views as the primary interface. This approach ensures that if users can edit a view, they can use this module without needing extensive technical expertise. Additionally, a widget is provided within the Views configuration to search for existing CSV files that have been uploaded to the site, further simplifying the process.

Regarding feedback from the Drupal community, Daniel noted that Views CSV Source is still relatively new, so he hasn't received extensive feedback from actual use. However, there has been significant appreciation for the module's capabilities. He shared a success story where he used the module on one of his projects to embed several charts within a long-form page, demonstrating its practical application and effectiveness.

Daniel also shared his perspective on the Drupal Starshot Initiative, a topic currently gaining significant attention within the community. He expressed optimism about its potential impact and the future of Drupal, particularly in making the platform more accessible to a broader range of users, including site-builders and those with low-code skills.

"I’m feeling fairly optimistic about it. The reinvigorated effort to bring site-builders and low-code users to Drupal appeals to me; I started this way, so maybe I’m biased, but I want Drupal to benefit the average person with big ambitions, not just enterprises. 

The initiative also gives a shot-in-the-arm to certain features like Recipes, Layout Builder, Project Browser, and Automatic Updates. I think the various ' flavours' of Drupal are going to be what excites people. I am interested in working toward what I’ll call 'Drupal as a Data System,' which combines Drupal’s superiority in content management with in-built data storage, management, and visualization features. Hopefully more on that to come!"

Daniel expressed his hope for a new generation of open-source contributors who, empowered by AI, will start coding and feel a responsibility to give back to the community. He also anticipates more sophisticated contributions from long-time contributors, such as improved testing and better code quality. He believes that with fewer excuses for leaving translations unfinished or providing poor documentation, these enhanced contributions could create a virtuous cycle within the community.

When advising newcomers to Drupal who want to develop modules or contribute to the community, Daniel emphasized the importance of engaging with the community.

"Start by becoming part of the community; create your Drupal.org profile, join the Slack workspace, find a local meet-up group, and start following other Drupal users on social media. You’ll start to absorb knowledge about how to make the best use of Drupal. 

You can learn a lot from participating too: writing an issue or a post requires you to think about what you did and what the circumstances were. Remember, there are almost always other people who are facing the same issues you are!"

From a code contribution perspective, Daniel encouraged new contributors to start with manageable projects. He suggested that they find an abandoned Drupal 7 module that could be useful to their work and try to update it to be compatible with Drupal 10/11. He mentioned that this task usually doesn't require advanced PHP skills and can be a valuable learning experience. 

Daniel advised that once the updated version is complete, contributors should release it and take responsibility for maintaining it. He shared that when he took over the Charts module, he didn't consider himself a programmer and received significant help from colleagues and the community. This experience forced him to continually learn and improve his PHP skills. Daniel also noted that AI tools can now assist with many tasks, further lowering barriers for new contributors.

Disclosure: This content is produced with the assistance of AI.

Note: The vision of this web portal is to help promote news and stories around the Drupal community and promote and celebrate the people and organizations in the community. We strive to create and distribute our content based on these content policy. If you see any omission/variation on this please let us know in the comments below and we will try to address the issue as best we can.

Related Organizations

Advertisement Here

Upcoming Events

Advertisement Here