Agile@School – Anno sesto, terzo incontro

Arriviamo oggi al terzo appuntamento di Agile@School 2021. Pier-Paolo ci racconta l’attività dal punto di vista della programmazione.

Negli incontri precedenti abbiamo cominciato a tracciare la strada delle attività (come vedremo, non tutte strettamente legate alla programmazione vera e propria) che i ragazzi dovranno svolgere per arrivare al loro obiettivo, cioè rilasciare un videogioco in stile libro-game.

Lungo questo percorso vogliamo insegnare loro non solo gli aspetti “tecnici” di realizzazione di un software, ma anche quello che sta “al contorno” di un progetto di questo genere: parliamo ovviamente dell’approccio Agile – DevOps, quindi tutta una serie di pratiche e cerimonie volte a facilitare la collaborazione, la condivisione delle conoscenze e la gestione dei progetti. È risultato evidente come sia già consolidato l’utilizzo degli stand-up meeting all’inizio di ogni lezione, tant’è che oggi sono stati i ragazzi stessi a proporsi per un veloce aggiornamento sullo stato dei progetti, dal quale sono emersi spunti molto interessanti legati agli aspetti di software selection di cui si era parlato nel post precedente.

Infatti, due dei gruppi che avevano scelto di utilizzare Unreal Engine per la realizzazione del loro progetto si sono resi conto che non avrebbero fatto in tempo a produrre qualcosa di funzionante entro la fine degli sprint: questo non li ha però scoraggiati e, messe da parte le loro preferenze personali, hanno avuto un buon senso di responsabilità nel decidere di intraprendere una strada di sviluppo differente, pur mantenendo intatta la trama principale scelta inizialmente.

Il Team Fisher ha quindi deciso di realizzare il videogioco tramite una web application, fruibile quindi da browser, con l’intenzione di integrare nel gioco dei file multimediali precostruiti. Il Team GentsAndLady-py ha invece deciso di implementare una console application in Java; anche loro hanno dichiarato di voler utilizzare grafica e audio per aumentare il coinvolgimento all’interno del gioco.

In entrambi i casi, tale decisione è stata presa non tanto in base alla difficoltà della creazione del software tramite lo stack tecnologico scelto, quanto alla valutazione delle tempistiche di rilascio del software stesso. Questa a mio avviso è stata una scelta molto oculata, soprattutto se considerata in un contesto aziendale in cui i tempi sono fondamentali forse più che la tecnologia scelta.

Dopo una rapida revisione delle board di ogni team è seguita una parte di suggerimenti e correzioni delle user story inserite dai ragazzi: la scrittura di questi item talvolta non è facile neanche per chi segue le pratiche Agile da molto tempo, per cui era facile aspettarsi qualche sbavatura. Ci aspettiamo che per la prossima lezione le user story vengano migliorate, e soprattutto che vengano utilizzati in maniera più intensiva i task per la suddivisione delle attività.

A questo proposito, una nota positiva viene dal Team Fisher che ha chiesto se fosse corretta la suddivisione delle attività da loro individuate fra più persone in parallelo e l’affiancamento di due persone su una stessa attività di programmazione: non solo questo è corretto, ma è proprio quello che si auspica con l’adozione delle pratiche Agile! E pur non sapendolo (non ne avevamo ancora parlato), ne hanno messo in evidenza uno degli aspetti chiave, cioè il pair programming. Molto bene!

Il resto della lezione è proseguito con l’introduzione del software per il controllo del codice sorgente Git, che però temo non abbia avuto l’impatto sperato. Purtroppo, non siamo stati in grado di effettuare degli esercizi live, un po’ perché non è stato possibile installare il software sulle postazioni usate dai ragazzi, un po’ per problemi logistici (noi e i ragazzi siamo lontani “fisicamente”, chi in classe, chi da casa, e la connessione instabile non permetteva una condivisione dello schermo). Abbiamo comunque chiesto a ciascuna squadra di creare un repository su GitHub, sia per poterlo connettere successivamente ad Azure DevOps e dimostrare come si possano collegare i task alle modifiche del codice, sia per iniziare a caricare i file del proprio progetto.

See the source image

Con questo appuntamento si concludeva in teoria il primo sprint, ma fra cambi di tecnologia e necessità di adattamento all’Agile i ragazzi non sono ancora riusciti produrre nulla di significativo, ma non disperiamo, c’è tutto il secondo sprint!

Agile@School – Anno sesto

Siamo ancora in pandemia. Abbiamo una buona percentuale di vaccinati, ma ancora dobbiamo avere a che fare con relazioni in remoto. Lo stesso vale per il nostro amato e inarrestabile progetto: Agile@School. Siamo arrivati al sesto anno consecutivo e questa volta, con un’idea orientata al mondo videoludico.

Non a caso l’immagine di copertina richiama uno dei giochi punta e clicca più conosciuti di tutti i tempi, col quale ho perso ore e ore nei pomeriggi della mia adolescenza: The Secret Of Monkey Island tm.

Il cosa

“Facciamo un gioco, sì, produciamolo. Pur semplice che sia, ma creiamo qualcosa di nostro”. Questa è la missione che i ragazzi hanno quest’anno. Il tipo è quello della saga di “Lupo Solitario” (Lone Wolf), un famoso libro game di qualche tempo fa. Abbiamo deciso di far fare agli studenti un piccolo motore basato su “decisioni –> conseguenze”, un grafo in cui una storia viene sviluppata e diventa la strada di un personaggio con una semplice caratteristica, l’energia vitale.

Il come

Decidano i ragazzi. C++? Java? Angular? Javascript? Python? Quello che vogliono. Ci sono solo tre vincoli:

  • La storia deve essere derivata dal programma di una materia, anche per coinvolgere altri professori.
  • La storia deve essere un’avventura semplice, per arrivare ad un lavorato funzionante in un solo mese.
  • Le scelte possibili devono essere al massimo 3, con altrettante conseguenze: avanti senza subire danni, avanti subendo danni, fine del gioco prematura (fine energia vitale improvvisa).

Gli strumenti

La gestione del progetto sarà in Agile, e con SCRUM. Per implementare la parte di gestione, abbiamo creato 4 team con altrettanti backlog, solo Product Backlog Item e Task. Utilizzeremo git come source control manager e, in generale, ogni editor di codice legato alla tecnologia scelta dai ragazzi.

Infine, le presentazioni prevedono un piccolo video, una serie di mock grafici nell’arco del progetto e una bella demo del software alla fine di tutto.

I team

I ragazzi hanno già scelto i nomi dei team:

  • FISHER: “perché abbiamo tutti un soprannome comune che richiama tale nome”
  • CROMOSOMI#: “un joke per il linguaggio C++”
  • MONKETECK: “due parole, e la seconda per esprimere senso di tecnologia”
  • GENTS&LADY.PY: “gentlemen e una ragazza, in python, o almeno ci proviamo”

Ogni team è di 5/6 persone e le figure che abbiamo identificato sono:

  • PO: product owner, uno per team, che si riferirà a me per lo stato dei lavori (non dimentichiamo che io sono il cliente del gioco).
  • Marketing: figura che curerà la presentazione.
  • Tech lead: figura che seguirà le implementazioni anche a livello decisionale, riferimento di Pier-Paolo, che anche quest’anno sarà il loro appoggio tecnico.
  • Developer: chiunque partecipi al progetto e sviluppa parti del gioco

Insomma, ci sarà da divertirsi. Aspettatevi qualche post nel breve, perché i ragazzi hanno già iniziato a buttarsi a capofitto nel progetto. La passione sembra averli abbracciati.

Stay Tuned!

Engage Stories – digest 1

Come promesso, ecco la prima tranche di #EngageStories:

Altre interessanti storie, sempre in Engage, qui:

Our 2021

Everyone was waiting for this 2021, 2020 has been one of the worst year ever due to the Coronavirus pandemic. We need to start from scratch, both for relationships and professional development.

Due to the COVID-19, I have lost relatives and also some of my relationships with my fellows. From a professional perspective, I’ve travelled less, worked mostly in remote-working and I’ve invested in making our daily work in our company smarter. Anyways, we’re speaking about two sides of the same coin: trouble Vs opportunity.

For tech companies, the crisis has been a great opportunity. The disease has triggered (forced to be honest) a set of unstoppable changes. This is true also for Countries’ governments, school, services in general. At the same time, many small businesses and craftsmen stopped working due to lack of consumers. The world is changed and is going to change more and more. The change is killing someone and is testing severely the resilience of the others.

Luckily, our company is still alive and we are growing. I’ve learned many lessons in terms of recruitment, team management, tech stuff, project management. I’ve started to change my mind about multi-tasking and I’ve switched from many things at “the same time” to “the right number of things well managed” (but not only one 😉 I’m not yet able to follow such pattern).

We have dealt with the second big step of scaling out. Our organization is becoming bigger and more structured. The model is still flat, but now each individual is in charge of a single role. As a result, the quality of the product is improved, people are more focused and committed, but on the other hand, the time to market has increased. We have added a new role, which orchestrates the projects into the roadmap, increased the number of POs, created an R&D department, and fully involved Operations guys into the lifecycle (thanks Wikis!). All this stuff has been the game-changer for us.

It’s been so difficult to learn and cover so many topics, but we’ve accomplished our mission. Now we’re ready for the next year with many ideas. We already know that our team will grow more, and we’ll require new approaches. We know that we will deal with teams in different timezones and languages, so what is “smart” now, must become “smarter” tomorrow. Exciting, for sure!

Why am I writing this post? Not just to wrap-up something that happened this year. I would share with you that, starting from February 2021, Engage Labs (this will be our new name) will publish our guys’ blog posts periodically (in Italian language). Our great team will cover many topics, like professional development, lessons learned in the company, technology, and so on. We are a DevOps oriented organization and we would like to let our guys describe themselves.

So, stay tuned😉

DevOps journeys series – Vertica release pipeline with Azure DevOps – Ep. 02 – build

In a previous post, we’ve described the “from scratch” approach on the development side. When everything works well there, a push (or check-in) triggers the build engine. We must deal with two SQL Server instances (SSIS Servers hereafter), with an environment for each of them:

The build pipeline

The SSIS Servers keep Vertica‘s test and production mappings as well as test and production connection strings for the SQL Server databases. So we need the right variable mapping for all the scenarios, but this is not in the scope of the post, we will speak about it in the next posts. Anyways, here is how the build pipeline works:

Our build process

You may notice that the task “Copy vertica deploy scripts” is disabled. Well, to be honest, right now we’re waiting for the target integration environment.

Build process explained

In the beginning, the build server gets the source files from the repository and creates the target artifacts folder with a Powershell script. This will be the path from which we will push the artifacts to the release pipeline.

The build server generates the .ispac file for the SQL Server Integration Services packages using the dedicated task. The copy tasks will be executed:

As you can see, we’ve got a set of utilities and transformation tools, that will be executed in the release pipeline as well as the environment script. This one contains the SSISDB variables mapping and the SSIS Project configurations statements. Misc files, .sql files for environments and the .ispac file will be copied to the target artifacts folder.

The tasks above copy our template of the .nuspec file to generate the NuGet file (NuGet pack step). This is what we get using NuGet:

Then, we’re ready to publish the files to the release pipeline. We will see how the release pipeline works in the next posts.

Ehm… you miss Vertica

Yes, you’re right. But, it’ll be just a copy of .sql files to the artifacts folder. We will see how the release manager will execute them, so…

Stay tuned!

Display Reporting Services usage statistics with Grafana

Introduction

In this post, we will describe an efficient way of showing the usage statistics of our SQL Server Reporting Services hosted reports. Most of the queries below have been addressed in another article published by Steve Stedman. Even though they are really useful, the article shows their results through SQL Server Management Studio.

The problem

One of the problems that often occur in our organization as well as some of our customers, is to get immediate feedback about usage statistics of reports. Usually, the request of creating reports is out of control and some of them are executed only “that time” and not anymore. In the worst-case scenario, many of them aren’t executed at all and some of them could become even overlapped or duplicated.

Therefore, it is important to know the usage statistics, user by user and report by report, to make the reader aware of them, let him interpreting the values of the same query in multiple ways and graphical layouts. While this is not possible with a tabular format (unless you export the values using any external tools such as Excel) it is simpler when it comes to a dashboard.

Our solution: Grafana

We considered two factors: simplicity and efficiency, in order to make this first-sight dashboard. Grafana enables us to get both of them, as well as being very powerful and immediate. Even though this is not the right definition for it, we can say that “it is a portal to create dashboards using connectors, which support the most famous tools that return data”. We can find them in its marketplace. For instance, tools such as PRTG and Prometheus (monitoring), NewRelic (APM), also SQL and NoSQL data sources are supported:

Obviously, we can find SQL Server. Also, we can contribute to create others, as well as to modify Grafana itself, since it is completely an Open Source project. Examples of possible graphical representations are listed below:

Creating a dashboard is really simple. Just add each panel with a button.

Then, write the query and modify settings to get the desired type of representation.

As mentioned before, the connectors are many. Once selected you can to configure them with parameters:

If you would like to install and configure Grafana you can read the official documentation which also includes a short guide that illustrates how to take your first steps.

That’s it!

Conclusions

With half a day of work (including the setup of the server), we have solved one of the most important problems of our customers, derived from the lack of awareness of reports deployed in production environments. We did it with very little effort and the result, as you can see, is pleasant and effective. Everything is now ready to be published every time we update the dashboards also through a delivery software (Octopus Deploy, Jenkins or Azure DevOps) so all these things fall into the second and third way of DevOps (according to The Phoenix Project): Immediate Feedback and Continuous Improvement.

Stay Tuned!

DevOps journeys series – Vertica release pipeline with Azure DevOps – Ep. 01 – development (part 2)

Intro

In the previous post we’ve described the idea behind the automation we’re trying to implement on a scenario based on MicroFocus Vertica database.

How it works

This “sandbox” is not a real isolated development workstation. Let’s separate it into two parts, the first one for the development on everything but Vertica (Windows local workstation) and the other one for a Vertica instance (probably Unix/Linux VM) shared between developers.

In this shared instance we will get a schema for each developer is working on the solution, in order to let everyone to get his own “environment”.

The source control folder tree (which will be TFVC source control on-premises) will be designed on the desired branch as the following:

/Project
    /Instance
        /Process1
            /_Master
                schema.ps1 
                tables.ps1
                views.ps1
            /Tables
                Table1.sql
                Table2.sql
            /Views
                View1.sql
                View2.sql
            Schema.sql
        /Process2
            /Tables ...
            /Views ...

As you ca see, under the Project folder there is the Vertica database folder, which contains, schema by schema, all the .sql files for Tables and Views DDLs (CREATE TABLE and CREATE VIEW). You can notice also .ps1 files, which contains the list of executions based on a certain order (business driven).

The file for a, let’s say, “Table1”, can be like this one:

CREATE TABLE :SCHEMA.Table1
(
    RowId int NOT NULL,
    RowStringValue varchar(30) NULL,
    CONSTRAINT PK_<schema>Table1 PRIMARY KEY (RowId)
);

We’ve added a :SCHEMA parameter, which allows each developer to create its own schema as described before. This is the only way we’ve found for isolating developers in a Vertica shared instance, avoiding an instance for each developer, which could be resource intensive for available PCs. Running the application locally, before committing any change set to the Source Control, a simple tool will execute .sql files with the new schema name and in the sort order given by the .ps1 file.

The Tables.ps1 file can be as the following:

param(
[parameter(Mandatory=$true)]$hostname,
[parameter(Mandatory=$true)]$port,
[parameter(Mandatory=$true)]$user,
[parameter(Mandatory=$true)]$psw,
[parameter(Mandatory=$true)]$schemaName,  
[parameter(Mandatory=$true)]$scriptsFolder
)

$schemaCommand = "vsql -h $hostname -p $port -U $user -w $psw -v SCHEMA=$schemaName -f $(Join-Path $scriptsFolder "Table1.sql")"
Invoke-Expression -command '$schemaCommand'

$schemaCommand = "vsql -h $hostname -p $port -U $user -w $psw -v SCHEMA=$schemaName -f $(Join-Path $scriptsFolder "Table2.sql")"
Invoke-Expression -command '$schemaCommand'

You may notice the term “vsql”, which is the command line provided by Vertica for executing queries. Further information here.

Also, usernames and passwords will be stored in an external config file (or a secured API), like the following one:

{
"host": "MyHost.Hos.Ext",
"port": 1234,
"user": "user",
"psw": "password",
"schemaName": "MYSCHEMA"
}

We’ve got the DDLs, the PoSh files for executing them and the Vertica command line. Good, in a development environment, however, a set of tools should be prepared for helping us to keep these artifacts on a single pipeline, too. This is the reason why we’ve created a “builder” script, like  this one:

$config = Get-Content (Join-Path $currentFolder "Build-Config.json") | Out-String | ConvertFrom-Json

$schemaCommand = $(Join-Path $scriptsFolder "Tables.ps1") 
$schemaCommand += " -hostname $($config.host)" 
$schemaCommand += " -user $($config.user)"
$schemaCommand += " -port $($config.port)"
$schemaCommand += " -psw '$($config.psw)'"
$schemaCommand += " -schemaName $($config.schemaName)"
$schemaCommand += " -scriptsFolder $scriptsFolder"

Invoke-Expression -command $schemaCommand

This is another layer of management, which allows us to organize every part of the DDLs to be executed against Vertica.

Note: Our scripts will destroy and rebuild any given SCHEMA. But this is the way we like.

Now, let’s see the possible scenarios.

Start from scratch or get started

When someone wants to start from scratch, this is the pipeline to follow:

  1. get latest version of the branch;
  2. check and change the configuration file (JSON);
  3. execute the create-vertica-database-from-scratch.bat file (it contains our powershell “build” script);
  4. that’s it, we’ve got a new schema in Vertica, empty and ready to be consumed.

If you want to preserve your data, this is not the right path for you. Executing the “builder” tool is optional.

New development

When a developer would make and try its changeset:

  1. change Visual Studio application (SSIS or SSRS here) when needed;
  2. change the Vertica schema (adding tables, columns and so on);
  3. get the .sql file of a new object or change the .sql file of an object which has been updated;
  4. replace them into the TFVC file structure;
  5. change the .ps1/.txt files if any DDL has been added (or if something that impacts on the order occurs);
  6. build the Visual Studio application and try it;
  7. When everything works good, check-in.

Now, everyone can get the latest changes in a CI way.

Get delta changes

When a developer is going to get the latest changes that contains an updated version of the Vertica objects, but wants to preserve its data, this is a little bit more tricky. The person who has made the change could share in a collaborative chat tool the ALTER script. This is not so reliable and comfortable, but without any comparison tool, there isn’t any best way to make this happen.

That being said, we’ve implemented our diff-script generator, based on the analysis of Vertica metadata (the catalog, browsing v_internal objects). So, after our friend gets the latest version, he executes a generate-diff-script.bat tool and lets this script to execute the generated ALTER script. Tricky, but it works like a charm (we will speak about this comparison tool in next posts, maybe) . Anyway, we’re looking forward hearing updates from MicroFocus. Hopefully, we’ll get an official diff tool from them soon!

Conclusions

I’ve just shown the way we’re managing tables DDLs and how we’ve created PowerShell scripts, but the real scenario is more complex. We have to drop Vertica schema (CASCADE DROP) before, then re-creating the new parametrized schema, then tables, then views and so on. Sometimes, we’ve got Vertica schemas which are referenced each other, so we have to create for everyone of them tables before, then views. The business logic is managed by how we write the “build” PowerShell script as well as the automated build process will.

Also the build process is not always “straight”. Some of the business processes need to be managed in a dedicated way. Cross reference will occur but this is the job that the “builder” will do. Both for the manual and the automated one. Moving responsibility to the build process allows us to feel more comfortable about our development solution.

Last, but not least, the manual-development-build process allows the developer to choose between re-create the database or not. They should not waste time in managing the things that a script can do repeatedly and efficiently. This is the reason why we kept somehow static the PowerShell instead of writing complicated business logic. Just adding rows of vsql invocation in the right order, that’s it.

Two tech events in Parma, the city of food

SQL Saturday Parma, six years in a row. DevOpsHeroes, four. Parma has been a great place to reach, also for technical events. I’ve started organizing the first SQL Saturday in my birthplace in November 2014. After two years I tried to create a brand-new event, when the DevOps culture started to grow and when the agile became strong. DevOpsHeroes was born in 2016, a month before the SQL Saturday event, again, in Parma. Why change? The audience has been great (more than 200 attendees), the feedbacks, too. People who come here look comfortable with everything. Then, thanks to the University, which has been the selected location, both the events are still growing.

Let’s go in deep with the events.

SQL Saturday Parma (2014-now)

SQL Saturdays, a great format by Professional Association of SQL Server (PASS), is a well-known event all around the world. You can find hundreds of them here. In Parma, the event is completely free, with no pre-conference. It’s located on the Campus of the University of Parma, in order to make also the students as well as the school aware of this kind of events. Unfortunately, the audience doesn’t gather them as does for the professionals, so I’m working to make a better relationship with them, too. About the audience, some numbers:

Event Attendees Track Speaker Feedback
SQL Sat 355 (2014) 128 3 14 4.46/5
SQL Sat 462 (2015) 157 3 18 4.20/5
SQL Sat 566 (2016) 150 3 18 4.48/5
SQL Sat 675 (2017) 210 4 24 4.47/5
SQL Sat 777 (2018) 230 4 24 4.72/5

In 2014 the sessions were driven by, let’s say, classic topics, like DBA, Development, BI. Starting from 2016 the coverage changed a lot. More data visualizations, more automation, more BI in the cloud, more cloud itself. 2017 has been the game-changer about NoSQL sessions (on Microsoft Azure), too. The latest edition of SQL Saturday Parma introduced the AI and this year we are struggling for selecting the right sessions from a bunch of 70 proposals (all over the world). September, 30 the Call for papers (available here) will close and if you are in the area on November 23, or if you want to come in Italy for a weekend of training on Microsoft Data Platform with friends, #sqlfamily and good food you are welcome!

The event is strongly supported by the Italian #sqlfamily, especially my friends in UGISS. A big thanks go to them.

DevOpsHeroes (2016-now)

Started as a one-shot event, this is a four-year-in-a-row one. Riding the wave of enthusiasm derived from the SQL Saturdays in Parma and thanks to my work experience, who moved meanwhile from DBA skills to Data DevOps and automation, this event has been a pleasant surprise, yet it doesn’t gather as many people as SQL Saturday does (SQL Saturdays has got also its noise and the PASS support). The event was born for spreading the DevOps culture, not just the tools. Tools were described there just to pull out the advantages of the culture, which must be “soaked up” before going deeper. So, the event was born for the culture. And this has been (and it still is) one of our mission.

The event is held typically one month before SQL Saturday in the Campus of the University of Parma. It gets more than 120 attendees and this year the organization is expecting more, due to the great sponsors which help the edition.

As you can see, behind the hood there are two main “helpers”. Engage IT Services, which is the company whose I’m a co-founder, and GetLatestVersion.it, a great Italian online community for DevOps and ALM technologies. The event is totally free, and it will get 18 sessions with 3 tracks. The topics will cover Technologies, Methodologies and use cases (or Experience sessions). The call for paper is already closed and we’re finishing the program of that Saturday, October 26.

Wrapping up

SQL Saturday Parma website: https://www.sqlsaturday.com/895/EventHome.aspx

Registration: https://www.sqlsaturday.com/895/registernow.aspx

DevOpsHeroes website: http://www.devops-heroes.net/

Registration: https://www.eventbrite.it/e/biglietti-devopsheroes-2019-66796826105

DevOps journeys series – Vertica release pipeline with Azure DevOps – Ep. 01 – development (part 1)

Intro

As a consultant, life could be difficult when it comes to manage platform like Vertica, a columnar RDBMS by MicroFocus. Speaking about DevOps, database management systems like this one neither is well supported by built-in tools nor by any third party suites.

In my experience, which is focused on the Microsoft SQL Server world, tools like the ones made by Redgate or ApexSQL, plays a crucial role when it comes to DevOps. Unfortunately, this time I cannot find any real help. I think that it’s an exciting task, but I’ve to be careful. It’s like “reinventig the wheel” for a car (Vertica) that didn’t realize that it needs wheels. Strong and reliable wheels.

Scenario

In the scenario I’m working on, the data warehouse is managed by a layer of business logic implemented within this platform. Starting from SQL Server database populated by the application layer, the data rows pass through SQL Server Integration Services ETL packages, which transform them onto Vertica repository, and they end into a Business Objects layer (the presentation layer). Just to get the big picture, see the following diagram:

bi-scenario-scenario

Our mission is to share DevOps knowledge, culture and tools in order to automate a set of processes which are managed manually, right now. Not a simple task, even if the guys I’m cooperating with are well-trained and powerful in tech skills. Additionally, they are enthusiast and ready to change (and we know how this is usually a trouble). We will get obstacles only from a technology perspective, actually.

After days of documentation and questions to our MicroFocus contacts, it looks like that Vertica gives us no built-in way or tool for getting an automated process during the phase of the schema comparison. Also third-party tools aren’t that good. We’re trying to dig more and more in order to find out something on the internet. However, everything it’s a “little bit” tricky. To be honest it seems that no one tried to do DevOps with Vertica, even though this is a very good platform for speeding up heavy queries on huge amount of data. Yes, we downloaded a couple of officially suggested tools, whose documentation says that we can compare from and to Vertica itself, but no one fits our scenario. No command lines, no direct integration, no script generation. If you think about automated pipelines, this can be a big problem to deal with. I think that this should be the foundation of a DevOps approach. In the end, it looks like we’re the the first team that is trying to invest efforts into that. It’s funny, though. Hopefully, someone with experience in Vertica DevOps who reads this post could help us!

(After digging more, I’ve found this article that describes almost the same scenario. I hope to reach out the author out).

Prerequisites

As a kick-off, we’ve moved into the prerequisites, starting from the most important thing: make a process that will facilitate the team’s workday, instead of wasting time due to bad choices. This process should be simple and automated.

Then, we’ve argued about the IDE used by developers, evaluating differences between before and after solutions, getting pro and cons of them all. In the end, we agreed on the following solution, development side:

  • Visual Studio 2017 will be the editor for SQL Server Integration services solutions. This team, “BI team” hereafter, is working on SSIS and is separated from the application development team, at least at the moment;
  • TFVC (Azure DevOps on-premises) will be the Version Control System since the BI team is already using it. We will think about a migration path to git then, but for now we’d like to avoid any distraction out of the scope;
  • every get latest from the source control must synchronize all the DDL scripts for Vertica as well as the SQL Server and SSIS solutions, in order to get a sandbox with source code and the local database provisioning scripts (most likely we’ll get a virtual machine with Unix and a Vertica instance, too);
  • each SSIS project must be deployed to the local SQL Server instance when running solutions into the sandbox;
  • when all the SQL Server Integration Services packages are deployed, the list of Vertica DDL must be executed in order to create the database from scratch;
  • Optionally, mock inserts (also from file) could be added to the database in order to get data to work on.

Our idea

As you can see, in this pipeline, we will get everything we need in our sandbox. Just a note, we can choose how to provision the instance of Vertica, in order to scale out resources from local workstations and also use another O.S. It’s up to the team.

A big picture of this process can be summarized as follows:

001

Conclusions

As we’ve seen, the big picture is ready to be implemented and also the team knows perfectly what are the goals. We already know what technologies will be involved and how to keep them all together in order to make our pipeline.

Unfortunately, we cannot find any tool that helps us, so we’re preparing to jot some lines of code down.