Sovware Data Mandiri - Big Data Infrastructure

#Project Overview

At Sovware Data Mandiri, I served as a Junior System Administrator responsible for designing, implementing, and maintaining a comprehensive Big Data infrastructure. This role involved architecting scalable distributed systems, building observability solutions, implementing modern CI/CD practices, and establishing secure authentication mechanisms. The infrastructure supports both on-premise and cloud environments, handling complex data processing and storage requirements for enterprise operations.

#Company & Role Context

Sovware Data Mandiri is an enterprise data solutions company based in Bandung, Indonesia. As a Junior System Administrator in a hybrid work environment, I was tasked with building and maintaining the technical foundation that enables the company's data processing and analytics capabilities.

The role encompassed:

• Infrastructure design and implementation
• System administration and operations
• DevOps and CI/CD pipeline management
• Security and authentication infrastructure
• Monitoring and observability
• Performance optimization and troubleshooting

#My Responsibilities

As the primary infrastructure engineer, my key responsibilities included:

• Designing Big Data architecture for scalability, high availability, and fault tolerance
• Deploying and managing distributed systems on Proxmox hypervisor
• Building and maintaining comprehensive monitoring and observability stacks
• Developing and maintaining CI/CD pipelines using GitLab
• Implementing centralized authentication and authorization systems
• Configuring secure VPN services for infrastructure connectivity
• Leading the modernization of deployment processes

#Big Data Architecture & Technology Stack

#Storage & Data Processing

The infrastructure leverages multiple technologies for distributed storage and processing:

• MinIO

  • S3-compatible object storage for data lakes
  • High-performance distributed storage

• Ceph

  • Distributed storage system for block and object storage
  • Provides redundancy and fault tolerance

• Apache Iceberg & Nessie

  • Open table format for data lakes
  • ACID transactions and time-travel queries

• Apache Parquet & Avro

  • Columnar and row-based data formats
  • Optimized for analytics and compression

#Query & Analytics Engines

• Dremio

  • Self-service analytics platform
  • Unified query engine across data sources

• Trino (formerly PrestoSQL)

  • Distributed SQL query engine
  • Multi-source querying capabilities

• Apache Hue

  • Web interface for data exploration
  • Query editor and visualization tools

#Coordination & Streaming

• Apache Zookeeper

  • Distributed coordination service
  • Cluster management and leader election

• Apache NiFi

  • Data routing and transformation
  • Real-time data flow management

#Data Visualization

• Superset

  • Modern data visualization and dashboarding
  • Self-service analytics platform

• Metabase

  • Business intelligence and analytics
  • User-friendly query builder

#Infrastructure & Virtualization

• Proxmox

  • Hypervisor for virtual machine management
  • KVM-based virtualization platform

• Docker

  • Containerization for application deployment
  • Consistent environments across infrastructure

#Networking & Load Balancing

• Nginx

  • Reverse proxy and web server
  • Load balancing and API gateway

• HAProxy

  • High-performance load balancer
  • Advanced routing and failover capabilities

• Keepalived

  • Virtual IP management
  • High availability for load balancers

#Database

• MariaDB Galera Cluster

  • Multi-master replication
  • Synchronous replication for consistency
  • Active-active clustering

#Observability & Monitoring Stack

I built a comprehensive observability solution to ensure system reliability and performance:

• Prometheus

  • Metrics collection and time-series database
  • Scraping metrics from all infrastructure components

• Grafana

  • Visualization and dashboarding
  • Real-time monitoring and alerting

• Thanos

  • Long-term metrics storage
  • Global query view across Prometheus instances

• Loki

  • Log aggregation and indexing
  • Centralized logging for all services

• Alloy

  • Unified observability agent
  • Metrics, logs, and traces collection

This stack provides:

• Real-time system performance monitoring
• Centralized log aggregation and analysis
• Historical metrics retention and analysis
• Alerting and incident response capabilities
• Distributed tracing for troubleshooting

#CI/CD Pipeline & Deployment Automation

#GitLab CI/CD Implementation

I designed and implemented a modern CI/CD pipeline using GitLab:

• GitLab Runner

  • Distributed CI/CD executor
  • Docker-based job execution

• GitLab Container Registry

  • Private container image repository
  • Secure artifact storage

• Automated Workflows

  • Build automation
  • Testing and validation
  • Containerized artifact production
  • Semantic versioning

#Deployment Process Modernization

Led the transition from manual deployment processes to automated pipelines:

• Before: Manual SCP/SFTP-based deployments and manual tagging
• After: Automated CI/CD with Semantic Versioning and Conventional Commits

Benefits achieved:

• Improved release consistency
• Reduced deployment errors
• Faster time-to-production
• Better version tracking and rollback capabilities
• Enhanced team collaboration

#Security & Authentication

#Single Sign-On (SSO) Implementation

Implemented a centralized authentication infrastructure using Keycloak:

• Keycloak

  • Identity and access management platform
  • OAuth 2.0 and OpenID Connect (OIDC) support
  • User federation and role-based access control

• OAuth 2.0 & OIDC

  • Secure authentication protocol
  • Token-based authorization
  • Integration with multiple applications

#VPN Services

Configured secure VPN services for infrastructure connectivity:

• L2TP (Layer 2 Tunneling Protocol)

  • Tunneling protocol for VPN
  • Secure remote access

• IPsec (Internet Protocol Security)

  • Encryption and authentication
  • Secure site-to-site connectivity

• OpenVPN

  • Open-source VPN solution
  • Flexible and secure remote access

• OpenSSL

  • Cryptographic toolkit
  • Certificate management and encryption

#Key Achievements

Throughout my tenure at Sovware Data Mandiri, I achieved:

• Designed and deployed a production-grade Big Data infrastructure supporting enterprise-scale operations
• Built a comprehensive observability stack enabling proactive monitoring and incident response
• Modernized deployment processes, reducing manual errors and improving consistency
• Implemented enterprise-grade security with centralized SSO and VPN services
• Established best practices for infrastructure as code and configuration management
• Enabled the team to scale infrastructure efficiently while maintaining reliability

#Challenges & Experience

Key challenges during this role included:

• Managing complexity of distributed systems across multiple technologies
• Ensuring high availability and fault tolerance in production environments
• Balancing performance optimization with resource constraints
• Coordinating infrastructure changes with minimal downtime
• Learning and mastering multiple enterprise technologies simultaneously
• Troubleshooting complex issues in distributed systems

These challenges provided invaluable experience in enterprise infrastructure management and problem-solving.

#What I Learned

Through this role at Sovware Data Mandiri, I gained extensive experience in:

• Designing and implementing enterprise-scale Big Data architectures
• Managing distributed systems and ensuring high availability
• Building comprehensive observability and monitoring solutions
• Implementing modern CI/CD practices and automation
• Security infrastructure including SSO and VPN services
• Infrastructure as code and configuration management
• Performance optimization and capacity planning
• Troubleshooting complex distributed system issues
• Leadership in modernizing legacy processes
• Working with cutting-edge data technologies

#Why This Project Matters

This role is significant in my career because it:

• Represents full-time position in infrastructure and DevOps
• Demonstrates my ability to design and implement enterprise-scale systems
• Shows my capability to lead technical modernization initiatives
• Reflects my expertise in observability and monitoring
• Combines infrastructure, security, and automation expertise
• Showcases my ability to work with complex, distributed technologies

#Conclusion

My tenure at Sovware Data Mandiri as a Junior System Administrator was transformative in developing my infrastructure and DevOps expertise. By designing and implementing a comprehensive Big Data infrastructure, building observability solutions, modernizing CI/CD practices, and establishing secure authentication systems, I gained practical experience in enterprise-scale system administration.

This role reinforced the importance of reliability, scalability, and observability in production systems—principles that continue to guide my approach to infrastructure and DevOps work today.