Python Automation Engineer - Vice President

Job Application for Python Automation Engineer - Vice President at iCapital

iCapital is looking to hire an experienced Python Developer with expertise in web scraping, automated portal interactions, and cloud-native deployment using AWS. The ideal candidate will have hands-on experience working with Playwright for browser automation, managing multi-factor authentication (MFA) flows, and deploying scalable scraping tasks via AWS Lambda and related services. This role will assist in architecting and building robust, secure, and scalable scraping solutions that interact with complex web applications and secured portals.

Assist with the design and implementation of advanced scraping solutions using Python, Playwright, and AWS services.

Automate interactions with JavaScript-heavy and authentication-secured websites, including handling MFA, CAPTCHAs, and session or token-based login flows.

Architect scraping pipelines using serverless AWS components such as Lambda, Step Functions, S3, CloudWatch, and Secrets Manager.

Build systems that scale to support high volumes of data extraction with fault tolerance, retries, and intelligent logging.

Integrate and manage complex workflows across multiple portals, APIs, and data sources.

Contribute to architectural decisions, tooling, and best practices.

Bachelor’s or master’s degree in computer science, software engineering, or related field, or equivalent professional experience

10+ years of experience in Python development, with a strong focus on automation and data extraction

Proven expertise in web scraping using tools like Playwright, Selenium, Scrapy, BeautifulSoup, and requests

In-depth experience handling multi-step authentication flows, including multi-factor authentication (MFA), CAPTCHA solving, and session or cookie management

Proficient in deploying and managing scraping workloads in AWS, particularly Lambda, S3, IAM, CloudWatch, and Secrets Manager

Experience with asynchronous programming, headless browsers, and JavaScript-rendered content

Solid understanding of web protocols (i.e. HTTP, HTTPS, cookies, headers), and the ability to reverse-engineer network calls and authentication mechanisms

Comfortable working with APIs, JSON/XML, and data transformation

Experience with CI or CD pipelines, Docker, and infrastructure as code (i.e. CloudFormation, Terraform)

Familiar with data pipeline orchestration tools such as Apache Airflow or AWS Step Functions

Prior experience scraping from secure or enterprise-level portals (i.e. financial, healthcare, legal)

Background in data engineering or ETL workflows is a plus

Exposure to Python testing frameworks and writing unit or integration tests

The base salary range for this role is $130,000 to $160,000. iCapital offers a compensation package which includes salary, equity for all full-time employees, and an annual performance bonus. Employees also receive a comprehensive benefits package that includes an employer matched retirement plan, genero...