Job Description : SDE I, II(Python)
About Us
VCCEdge, part of Mosaic Digital is an upcoming company dedicated to building innovative products for private market stakeholders. Our platform provides valuable insights, analytics, and solutions to empower decision-makers to navigate the dynamic landscape of the private market.
Position : Software Development Engineer I, II(Python)
Overview
We are seeking talented Software Developers proficient in Python, with a specialization in data engineering and web crawling. This role entails the design and implementation of resilient data solutions, coupled with the development of efficient web crawlers for extracting information from diverse online sources. The ideal candidate will exhibit a passion for utilizing open-source libraries to gather, process, and analyze data.
Responsibilities
- Collaborate with product managers, cross-functional teams, and stakeholders to comprehend data requirements. Translate requirements into technical specifications, ensuring data quality and integrity.
- Design and implement data engineering solutions using Python to manage large and diverse datasets. Ensure efficient storage, retrieval, and processing of data, following best practices.
- Develop and maintain web crawlers in Python for extracting data from diverse online platforms.
- Implement scalable and robust data collection pipelines.
- Create scripts for data cleaning, transformation, and normalization, ensuring high-quality data.
- Design and implement data models, databases, and processing components for optimal performance.
- Implement algorithms to extract meaningful insights from raw data.
- Integrate with external APIs to enrich data and enhance overall information retrieval processes.
- Develop automation scripts to schedule and optimize web crawling and data processing workflows.
- Establish monitoring tools to track the health and performance of web crawlers and data processing systems
- Identify and address performance bottlenecks, ensuring optimal speed and responsiveness.
- Collaborate with cross-functional teams to seamlessly integrate data into larger systems.
- Develop and execute comprehensive test plans for features and data-driven functionalities.
- Stay updated with emerging technologies and contribute to the continuous improvement of development processes.
Requirements
- Bachelor's degree in computer science, Engineering, or a related field with 1 +years of experience.
- Proven experience as a Software Developer, showcasing a strong focus on Python, MongoDB, MySQL, Elasticsearch, Redis, and related technologies
- Expertise in data engineering, encompassing efficient data storage, processing, transformation, and retrieval of large datasets.
- Proficiency in web crawling using Python libraries and frameworks such as BeautifulSoup, Scrapy. Libraries like Pandas and NumPy for data processing and manipulation
- Experience in Restful API development and integration
- Familiarity with cloud platforms, such as AWS and Azure.
- Strong problem-solving skills with meticulous attention to detail
- Ability to work collaboratively in a dynamic development environment
(ref:hirist.tech)
