Refactor

[angoca]

Hi, I have been interested in this project for many years, and I have used in different occasions.
Currently, I have used this project for a class I give, to compare different RDBMS and test performance. I tried to use the version 5, but this requires many modifications in the code for each DB. I haven't tried version 6 yet, but I see the lack of documentation for extension to other RDBMS which makes this version very difficult to use for a student that is just learning database. I do not want them to port the application, I just want they focus in the database side.

I have done a big refactoring for version 4, which is available at: https://github.com/ECI-SGBD/BenchmarkSQL-4

I propose you I could do the same refactoring in version 6, in order to update some old dependencies this project. The changes I propose are:

• Use Maven instead of Ant.
• Include drivers in Maven's pom, instead of manual downloading o including in this code.
• Use packages for all classes.
• Use log4j v2, instead of log4j v1.
• Change System.output to loggers, for all classes (the 3 main methods: sql, local and benchmark).
• Organize files according Maven structure. This refers to resources directory, and a directory for each kind of file: r, python, Bash scripts, etc. This organize the run directory.
• All documentation in one directory, separated from code or scripts.
• Documentation for each RDBMS, not just one for all of them.
• Format the code according Eclipse format, which is a standard one.
• Organize imports.
• Use PMD, findbugs and checkstyle for better code.
• Create javadoc headers, in order to be capable of generating a documentation of the code.
• Put a FIXME or TODO, in each part of the code where an extension should be done for other RDBMS.
• Use Wiki, to explain things about this project. How to use the python script, generation of the diagrams with r, etc.
  • Documentation about the parameters that receives the application.
• Scripts to run in Windows.
• Scripts to run based on the Maven structure (target directory).
• Extra parameters, like:
  • schema, which is useful for databases like MySQL that do not use schemas. Or for other databases which uses default schema. This do not force to use benchmarksql schema.
  • Statement terminator, when ; is not available. This could use another one like 'GO' or '@', or even EOL.

If you agree with some of these changes, I could do them, and then I will create a pull request. In fact, I do not want to do a big refactoring in a program, and not being integrated into the master.

Please tell me which ones are you interested in.

[wieck]

On 3/28/21 3:50 PM, Andres Gomez Casanova wrote: Hi, I have been interested in this project for many years, and I have used in different occasions.

Hi, thank you for your interest. I haven't found much time to work on this project lately, but definitely want to come back and finish/improve things.

Currently, I have used this project for a class I give, to compare different RDBMS and test performance. I tried to use the version 5, but this requires many modifications in the code for each DB. I haven't tried version 6 yet, but I see the lack of documentation for extension to other RDBMS which makes this version very difficult to use for a student that is just learning database. I do not want them to port the application, I just want they focus in the database side.

Version 6 is a major rewrite of the main driver in that it separates the simulated users from the simulated application threads by use of terminal object queues. This allows BenchmarkSQL to actually measure the end user experienced system response time instead of the database transaction latency. So whatever we are going to do, it needs to be based on version 6. It now scales very well to thousands of warehouses, using the proper number of simulated terminals as well as keying and thinking time delays, without overwhelming the benchmark driver itself or requiring tens of thousands of database connections to do that. Extending version 6 to another DBMS is also completely different from previous versions. It sure could use more documentation though. What I have been experimenting with last is wrapping BenchmarkSQL itself into a Flask based Web-UI and then packaging it all into a Docker container. That will make it very easy to deploy it for example into cloud environments. A major remaining problem is the analysis of the per-transaction results and generating the graphs. Installing R in the Docker container bloats the image to about 1.5GB because it pulls in most of X11. And then R can easily run out of memory when trying to aggregate a longer benchmark run, like over a week or two with a decent server and nearly maxing it out. To improve that it might be necessary to collect those results in a pre-aggregated form and switch from R to numpy or something else. Regards, Jan

…

-- Jan Wieck Principle Database Engineer Amazon Web Services

[luss]

I'm very interested in Andres proposed modernizations.

…

On Mon, Mar 29, 2021 at 11:50 AM Jan Wieck ***@***.***> wrote: On 3/28/21 3:50 PM, Andres Gomez Casanova wrote: > Hi, I have been interested in this project for many years, and I have > used in different occasions. Hi, thank you for your interest. I haven't found much time to work on this project lately, but definitely want to come back and finish/improve things. > Currently, I have used this project for a class I give, to compare > different RDBMS and test performance. I tried to use the version 5, but > this requires many modifications in the code for each DB. I haven't > tried version 6 yet, but I see the lack of documentation for extension > to other RDBMS which makes this version very difficult to use for a > student that is just learning database. I do not want them to port the > application, I just want they focus in the database side. Version 6 is a major rewrite of the main driver in that it separates the simulated users from the simulated application threads by use of terminal object queues. This allows BenchmarkSQL to actually measure the end user experienced system response time instead of the database transaction latency. So whatever we are going to do, it needs to be based on version 6. It now scales very well to thousands of warehouses, using the proper number of simulated terminals as well as keying and thinking time delays, without overwhelming the benchmark driver itself or requiring tens of thousands of database connections to do that. Extending version 6 to another DBMS is also completely different from previous versions. It sure could use more documentation though. What I have been experimenting with last is wrapping BenchmarkSQL itself into a Flask based Web-UI and then packaging it all into a Docker container. That will make it very easy to deploy it for example into cloud environments. A major remaining problem is the analysis of the per-transaction results and generating the graphs. Installing R in the Docker container bloats the image to about 1.5GB because it pulls in most of X11. And then R can easily run out of memory when trying to aggregate a longer benchmark run, like over a week or two with a decent server and nearly maxing it out. To improve that it might be necessary to collect those results in a pre-aggregated form and switch from R to numpy or something else. Regards, Jan -- Jan Wieck Principle Database Engineer Amazon Web Services — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMWOHSRPRBVX57QY6P4RM3TGCOV7ANCNFSM4Z6LRPBQ> .

[wieck]

On 3/29/21 2:38 PM, Denis Lussier wrote: I'm very interested in Andres proposed modernizations.

I am interested too. But any changes need to be based on current master, which isn't completely up to date yet. I may get to it this weekend and report back where on the V6 path we are. Regards, Jan

…

On Mon, Mar 29, 2021 at 11:50 AM Jan Wieck ***@***.***> wrote: > On 3/28/21 3:50 PM, Andres Gomez Casanova wrote: > > Hi, I have been interested in this project for many years, and I have > > used in different occasions. > > Hi, thank you for your interest. > > I haven't found much time to work on this project lately, but definitely > want to come back and finish/improve things. > > > Currently, I have used this project for a class I give, to compare > > different RDBMS and test performance. I tried to use the version 5, but > > this requires many modifications in the code for each DB. I haven't > > tried version 6 yet, but I see the lack of documentation for extension > > to other RDBMS which makes this version very difficult to use for a > > student that is just learning database. I do not want them to port the > > application, I just want they focus in the database side. > > Version 6 is a major rewrite of the main driver in that it separates the > simulated users from the simulated application threads by use of > terminal object queues. This allows BenchmarkSQL to actually measure the > end user experienced system response time instead of the database > transaction latency. So whatever we are going to do, it needs to be > based on version 6. > > It now scales very well to thousands of warehouses, using the proper > number of simulated terminals as well as keying and thinking time > delays, without overwhelming the benchmark driver itself or requiring > tens of thousands of database connections to do that. > > Extending version 6 to another DBMS is also completely different from > previous versions. It sure could use more documentation though. > > What I have been experimenting with last is wrapping BenchmarkSQL itself > into a Flask based Web-UI and then packaging it all into a Docker > container. That will make it very easy to deploy it for example into > cloud environments. > > A major remaining problem is the analysis of the per-transaction results > and generating the graphs. Installing R in the Docker container bloats > the image to about 1.5GB because it pulls in most of X11. And then R can > easily run out of memory when trying to aggregate a longer benchmark > run, like over a week or two with a decent server and nearly maxing it > out. To improve that it might be necessary to collect those results in a > pre-aggregated form and switch from R to numpy or something else. > > > Regards, Jan > > -- > Jan Wieck > Principle Database Engineer > Amazon Web Services > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#3 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAMWOHSRPRBVX57QY6P4RM3TGCOV7ANCNFSM4Z6LRPBQ> > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACYRMQNAGVWNYMRHYX243DTGDCKNANCNFSM4Z6LRPBQ>.

-- Jan Wieck Principle Database Engineer Amazon Web Services

[wieck]

On 3/30/21 10:07 AM, Jan Wieck wrote: On 3/29/21 2:38 PM, Denis Lussier wrote: > I'm very interested in Andres proposed modernizations. I am interested too. But any changes need to be based on current master, which isn't completely up to date yet. I may get to it this weekend and report back where on the V6 path we are.

I checked the repositories against what I have locally. All is in sync. The current status in the master branch is: * The new V6 benchmark driver is fully functional and measures end user experienced SUT response times correctly. * The benchmark can be used stand alone (the old fashioned way) as well as inside the POC Flask UI. * The POC Flask UI can be used inside and outside of Docker. That means we are good with respect to functionality for now. There are plenty of work items left to complete V6: * Replace the report generator with a matplotlib implementation. This will shrink the Docker image in half! * Make sure that the new report generator is capable of handling a couple hundred GB of transaction result data. * Implement an actual Flask UI. What is there now is really just a proof of concept. We need a real implementation with menus and server/configuration management. * Document how to use all of that with/without Flask/Docker. The only critical one I see is the report generator. So I'm going to have a look at the matplotlib stuff. The main problem with the R based report is that it needs to load the same CSV file into memory over and over. And those result CSV files get really big. I can generate a 9GB result CSV on my laptop in just a day. Let that benchmark run against a serious server for a week or so and you need a machine, larger than the DB server, to process the result data. If we can't get the "real" Flask UI done for V6, I'm fine with going with the current POC. It works, although it is ugly. Just like me. All that said, I think our overall plan should be: 1) Get V6 into a shape that really scales to large servers and long run times. Disk space should not be a problem, memory is. 2) Get V6 released (requires doc work). 2.5) Breathe 3) Restructure and do the improvements, that Andres proposed. 4) Release all that (and whatever else we can do) as V7. 5) Rinse, repeat. Did I miss anything? Regards, Jan

…

-- Jan Wieck Principle Database Engineer Amazon Web Services

[luss]

We are excited to accept code modernizations for v6 from the community. Also, Jan is being shy... he is a handsome man.

…

On Tue, Mar 30, 2021 at 10:35 PM Jan Wieck ***@***.***> wrote: On 3/30/21 10:07 AM, Jan Wieck wrote: > On 3/29/21 2:38 PM, Denis Lussier wrote: >> I'm very interested in Andres proposed modernizations. > > I am interested too. But any changes need to be based on current master, > which isn't completely up to date yet. I may get to it this weekend and > report back where on the V6 path we are. I checked the repositories against what I have locally. All is in sync. The current status in the master branch is: * The new V6 benchmark driver is fully functional and measures end user experienced SUT response times correctly. * The benchmark can be used stand alone (the old fashioned way) as well as inside the POC Flask UI. * The POC Flask UI can be used inside and outside of Docker. That means we are good with respect to functionality for now. There are plenty of work items left to complete V6: * Replace the report generator with a matplotlib implementation. This will shrink the Docker image in half! * Make sure that the new report generator is capable of handling a couple hundred GB of transaction result data. * Implement an actual Flask UI. What is there now is really just a proof of concept. We need a real implementation with menus and server/configuration management. * Document how to use all of that with/without Flask/Docker. The only critical one I see is the report generator. So I'm going to have a look at the matplotlib stuff. The main problem with the R based report is that it needs to load the same CSV file into memory over and over. And those result CSV files get really big. I can generate a 9GB result CSV on my laptop in just a day. Let that benchmark run against a serious server for a week or so and you need a machine, larger than the DB server, to process the result data. If we can't get the "real" Flask UI done for V6, I'm fine with going with the current POC. It works, although it is ugly. Just like me. All that said, I think our overall plan should be: 1) Get V6 into a shape that really scales to large servers and long run times. Disk space should not be a problem, memory is. 2) Get V6 released (requires doc work). 2.5) Breathe 3) Restructure and do the improvements, that Andres proposed. 4) Release all that (and whatever else we can do) as V7. 5) Rinse, repeat. Did I miss anything? Regards, Jan -- Jan Wieck Principle Database Engineer Amazon Web Services — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMWOHXSJBMUYZVLVUEWQJ3TGKDADANCNFSM4Z6LRPBQ> .

[wieck]

On 4/1/21 9:02 AM, Denis Lussier wrote: We are excited to accept code modernizations for v6 from the community.

There is one more very important issue. The new Oracle drivers actually have deprecated the Oracle ARRAY, which is why we currently need the driver at compile time for Oracle stored procedure support. The recommendation is to convert everything to use the generic java.sql.Array interface instead. This is a must do for V6 because it will allow us to ship a prebuilt Docker container and the user then can supply the vendor specific JDBC drivers in an external directory at runtime. There will be more like that. V6 currently can also runs the generic code (no stored procs yet) against Microsoft SQL Server. Regards, Jan

…

-- Jan Wieck Principle Database Engineer Amazon Web Services

[angoca]

Hi guys,

The code modernizations that I propose do not change the current functionality, and this won't correct or add new functionality. It just will use new frameworks that eases the deployment. The changes I propose will be done in parts, in order to have a tracking of the evolution; not just a big change.
The important thing to take into account with this refactoring is that the firsts "pull request" will have a lot of visual changes:

• The files will be moved to other directories to comply with Maven structure.
• The shell scripts will be modified to support Maven structure in the target directory instead of the run directory.
• Libraries like drivers will be automatically downloaded and included in the target directory by Maven.
• All documentation in one directory, separating it from code or scripts.

I will adjust the scripts to work with Postgres; currently I cannot test against Oracle.

Once files are moved, then the entire code will be adjusted.

• The code will be formatted to Eclipse syntax format.
• The imports will be organized.
• Packages will be used for all classes.

You can clone the pull request into a new branch, or locally, test what I propose, and if you agree, then perform the Pull request.

Once, I have done that, the basic structure will be fine to do these improvements progressively.

• Use of log4j v2.
• Configuration of log4j with packages.
• Scripts to run in Windows.
• Change System.output to loggers, for all classes (the 3 main methods: sql, local and benchmark).
• Documentation for each RDBMS, not just one for all of them.
• Modification of HOW TO RUN files to comply with Markdown syntax.
• Put a FIXME or TODO, in each part of the code where an extension should be done for other RDBMS.
• Use Wiki, to explain things about this project. How to use the python script, generation of the diagrams with r, etc.
  • In the wiki, there could be a page per RDBMS to explain the details and points of performance improvement.
• Create javadoc headers, in order to be capable of generating a documentation of the code.
• Documentation about the parameters that receives the application.
• Extra parameters, like:
  • schema, which is useful for databases like MySQL that do not use schemas. Or for other databases which uses default schema. This do not force to use benchmarksql schema.
  • Statement terminator, when ; is not available. This could use another one like 'GO' or '@', or even EOL.
• Use PMD, findbugs and checkstyle for better code.
• Use of Travis-CI to do Continuous Integration.
• Generation of a GitHub web page, based on the files under doc.
• Include support to other RDBMS, starting with Db2, and then extending to others.

Finally, I have theses questions, I need your answer to proceed:

• It is a good practice to have packages for all classes. I propose to use the following package name base, that easily reference the class to the hosting facility of the code:

  com.github.pgsql-io.benchmarksql

Under this package, all classes will be placed under different sub packages (jtpcc, loader, etc.)

• Which minimum java version BenchmarkSQL should support?
  • Java 1.7 is not longer maintained, but very popular.
  • Java 8 is still maintained and very popular.
  • Any older one?
  • Any newer, like java 9 with Lambda expressions?

[luss]

Sounds good to me. We just want to be sure that the pull request you submit are to https://github.com/pgsql-io/banchmark on the Master branch.

…

--Luss

On Thu, Apr 1, 2021 at 11:55 AM Andres Gomez Casanova < ***@***.***> wrote: Hi guys, The code modernizations that I propose do not change the current functionality, and this won't correct or add new functionality. It just will use new frameworks that eases the deployment. The changes I propose will be done in parts, in order to have a tracking of the evolution; not just a big change. The important thing to take into account with this refactoring is that the firsts "pull request" will have a lot of visual changes: - The files will be moved to other directories to comply with Maven structure. - The shell scripts will be modified to support Maven structure in the target directory instead of the run directory. - Libraries like drivers will be automatically downloaded and included in the target directory by Maven. - All documentation in one directory, separating it from code or scripts. I will adjust the scripts to work with Postgres; currently I cannot test against Oracle. Once files are moved, then the entire code will be adjusted. - The code will be formatted to Eclipse syntax format. - The imports will be organized. - Packages will be used for all classes. You can clone the pull request into a new branch, or locally, test what I propose, and if you agree, then perform the Pull request. Once, I have done that, the basic structure will be fine to do these improvements progressively. - Use of log4j v2. - Configuration of log4j with packages. - Scripts to run in Windows. - Change System.output to loggers, for all classes (the 3 main methods: sql, local and benchmark). - Documentation for each RDBMS, not just one for all of them. - Modification of HOW TO RUN files to comply with Markdown syntax. - Put a FIXME or TODO, in each part of the code where an extension should be done for other RDBMS. - Use Wiki, to explain things about this project. How to use the python script, generation of the diagrams with r, etc. - In the wiki, there could be a page per RDBMS to explain the details and points of performance improvement. - Create javadoc headers, in order to be capable of generating a documentation of the code. - Documentation about the parameters that receives the application. - Extra parameters, like: - schema, which is useful for databases like MySQL that do not use schemas. Or for other databases which uses default schema. This do not force to use benchmarksql schema. - Statement terminator, when ; is not available. This could use another one like 'GO' or '@', or even EOL. - Use PMD, findbugs and checkstyle for better code. - Use of Travis-CI to do Continuous Integration. - Generation of a GitHub web page, based on the files under doc. - Include support to other RDBMS, starting with Db2, and then extending to others. Finally, I have theses questions, I need your answer to proceed: - It is a good practice to have packages for all classes. I propose to use the following package name base, that easily reference the class to the hosting facility of the code: com.github.pgsql-io.benchmarksql Under this package, all classes will be placed under different sub packages (jtpcc, loader, etc.) - Which minimum java version BenchmarkSQL should support? - Java 1.7 is not longer maintained, but very popular. - Java 8 is still maintained and very popular. - Any older one? - Any newer, like java 9 with Lambda expressions? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMWOHRQ22CDN77UPPJHE7TTGSJQVANCNFSM4Z6LRPBQ> .

[luss]

Personally, I think we should support JDK 8, and JDK 11. Those are the two LTS releases AFAIK. On Thu, Apr 1, 2021 at 12:05 PM Denis Lussier ***@***.***> wrote:

…

Sounds good to me. We just want to be sure that the pull request you submit are to https://github.com/pgsql-io/banchmark on the Master branch. --Luss On Thu, Apr 1, 2021 at 11:55 AM Andres Gomez Casanova < ***@***.***> wrote: > Hi guys, > > The code modernizations that I propose do not change the current > functionality, and this won't correct or add new functionality. It just > will use new frameworks that eases the deployment. The changes I propose > will be done in parts, in order to have a tracking of the evolution; not > just a big change. > The important thing to take into account with this refactoring is that > the firsts "pull request" will have a lot of visual changes: > > - The files will be moved to other directories to comply with Maven > structure. > - The shell scripts will be modified to support Maven structure in > the target directory instead of the run directory. > - Libraries like drivers will be automatically downloaded and > included in the target directory by Maven. > - All documentation in one directory, separating it from code or > scripts. > > I will adjust the scripts to work with Postgres; currently I cannot test > against Oracle. > > Once files are moved, then the entire code will be adjusted. > > - The code will be formatted to Eclipse syntax format. > - The imports will be organized. > - Packages will be used for all classes. > > You can clone the pull request into a new branch, or locally, test what I > propose, and if you agree, then perform the Pull request. > > Once, I have done that, the basic structure will be fine to do these > improvements progressively. > > - Use of log4j v2. > - Configuration of log4j with packages. > - Scripts to run in Windows. > - Change System.output to loggers, for all classes (the 3 main > methods: sql, local and benchmark). > - Documentation for each RDBMS, not just one for all of them. > - Modification of HOW TO RUN files to comply with Markdown syntax. > - Put a FIXME or TODO, in each part of the code where an extension > should be done for other RDBMS. > - Use Wiki, to explain things about this project. How to use the > python script, generation of the diagrams with r, etc. > - In the wiki, there could be a page per RDBMS to explain the > details and points of performance improvement. > - Create javadoc headers, in order to be capable of generating a > documentation of the code. > - Documentation about the parameters that receives the application. > - Extra parameters, like: > - schema, which is useful for databases like MySQL that do not use > schemas. Or for other databases which uses default schema. This do not > force to use benchmarksql schema. > - Statement terminator, when ; is not available. This could use > another one like 'GO' or '@', or even EOL. > - Use PMD, findbugs and checkstyle for better code. > - Use of Travis-CI to do Continuous Integration. > - Generation of a GitHub web page, based on the files under doc. > - Include support to other RDBMS, starting with Db2, and then > extending to others. > > Finally, I have theses questions, I need your answer to proceed: > > - > > It is a good practice to have packages for all classes. I propose to > use the following package name base, that easily reference the class to the > hosting facility of the code: > > com.github.pgsql-io.benchmarksql > > Under this package, all classes will be placed under different sub > packages (jtpcc, loader, etc.) > > - Which minimum java version BenchmarkSQL should support? > - Java 1.7 is not longer maintained, but very popular. > - Java 8 is still maintained and very popular. > - Any older one? > - Any newer, like java 9 with Lambda expressions? > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#3 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAMWOHRQ22CDN77UPPJHE7TTGSJQVANCNFSM4Z6LRPBQ> > . >

[wieck]

On 4/1/21 12:06 PM, Denis Lussier wrote: Personally, I think we should support JDK 8, and JDK 11. Those are the two LTS releases AFAIK.

I am currently using java-1.8.0-openjdk to build and run V6 (master). I have not tried 11 yet. Will do that once I get my new server. Regards, Jan

…

On Thu, Apr 1, 2021 at 12:05 PM Denis Lussier ***@***.***> wrote: > Sounds good to me. We just want to be sure that the pull request you > submit are to https://github.com/pgsql-io/banchmark on the Master branch. > > --Luss > > On Thu, Apr 1, 2021 at 11:55 AM Andres Gomez Casanova < > ***@***.***> wrote: > >> Hi guys, >> >> The code modernizations that I propose do not change the current >> functionality, and this won't correct or add new functionality. It just >> will use new frameworks that eases the deployment. The changes I propose >> will be done in parts, in order to have a tracking of the evolution; not >> just a big change. >> The important thing to take into account with this refactoring is that >> the firsts "pull request" will have a lot of visual changes: >> >> - The files will be moved to other directories to comply with Maven >> structure. >> - The shell scripts will be modified to support Maven structure in >> the target directory instead of the run directory. >> - Libraries like drivers will be automatically downloaded and >> included in the target directory by Maven. >> - All documentation in one directory, separating it from code or >> scripts. >> >> I will adjust the scripts to work with Postgres; currently I cannot test >> against Oracle. >> >> Once files are moved, then the entire code will be adjusted. >> >> - The code will be formatted to Eclipse syntax format. >> - The imports will be organized. >> - Packages will be used for all classes. >> >> You can clone the pull request into a new branch, or locally, test what I >> propose, and if you agree, then perform the Pull request. >> >> Once, I have done that, the basic structure will be fine to do these >> improvements progressively. >> >> - Use of log4j v2. >> - Configuration of log4j with packages. >> - Scripts to run in Windows. >> - Change System.output to loggers, for all classes (the 3 main >> methods: sql, local and benchmark). >> - Documentation for each RDBMS, not just one for all of them. >> - Modification of HOW TO RUN files to comply with Markdown syntax. >> - Put a FIXME or TODO, in each part of the code where an extension >> should be done for other RDBMS. >> - Use Wiki, to explain things about this project. How to use the >> python script, generation of the diagrams with r, etc. >> - In the wiki, there could be a page per RDBMS to explain the >> details and points of performance improvement. >> - Create javadoc headers, in order to be capable of generating a >> documentation of the code. >> - Documentation about the parameters that receives the application. >> - Extra parameters, like: >> - schema, which is useful for databases like MySQL that do not use >> schemas. Or for other databases which uses default schema. This do not >> force to use benchmarksql schema. >> - Statement terminator, when ; is not available. This could use >> another one like 'GO' or '@', or even EOL. >> - Use PMD, findbugs and checkstyle for better code. >> - Use of Travis-CI to do Continuous Integration. >> - Generation of a GitHub web page, based on the files under doc. >> - Include support to other RDBMS, starting with Db2, and then >> extending to others. >> >> Finally, I have theses questions, I need your answer to proceed: >> >> - >> >> It is a good practice to have packages for all classes. I propose to >> use the following package name base, that easily reference the class to the >> hosting facility of the code: >> >> com.github.pgsql-io.benchmarksql >> >> Under this package, all classes will be placed under different sub >> packages (jtpcc, loader, etc.) >> >> - Which minimum java version BenchmarkSQL should support? >> - Java 1.7 is not longer maintained, but very popular. >> - Java 8 is still maintained and very popular. >> - Any older one? >> - Any newer, like java 9 with Lambda expressions? >> >> — >> You are receiving this because you commented. >> Reply to this email directly, view it on GitHub >> <#3 (comment)>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/AAMWOHRQ22CDN77UPPJHE7TTGSJQVANCNFSM4Z6LRPBQ> >> . >> > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACYRMWHDUCMNFF43KXQ3L3TGSKY7ANCNFSM4Z6LRPBQ>.

-- Jan Wieck Principle Database Engineer Amazon Web Services

[wieck]

I suggest you make yourself familiar with the current master branch of benchmarksql. There have been substantial changes to some of what you are touching below. More comments inline.

On 4/1/21 11:55 AM, Andres Gomez Casanova wrote: Hi guys, The code modernizations that I propose do not change the current functionality, and this won't correct or add new functionality. It just will use new frameworks that eases the deployment. The changes I propose will be done in parts, in order to have a tracking of the evolution; not just a big change. The important thing to take into account with this refactoring is that the firsts "pull request" will have a lot of visual changes: * The files will be moved to other directories to comply with Maven structure. * The shell scripts will be modified to support Maven structure in the |target| directory instead of the |run| directory.

New directory structure is fine.

* Libraries like drivers will be automatically downloaded and included in the target directory by Maven.

This I am very interested in. I was looking at the Maven Repository at https://repo1.maven.org/maven2 earlier and noticed that the Oracle and Microsoft JDBC drivers are there. Oracle even points to that repository on their download page. Which means that Oracle has changed their policy and no longer requires license click through for download. Bravo! I still don't think we should ship them or add copies of them to our repository, but if they are automatically downloaded at build time and if a user has an easy way to get them into a local lib directory to run a pre-built Docker container, that would be perfect.

* All documentation in one directory, separating it from code or scripts. I will adjust the scripts to work with Postgres; currently I cannot test against Oracle.

We definitely need to make sure that the changes do not interfere with the support for Oracle, MariaDB or MSSQL-server. Currently I am testing Oracle against an Oracle-11g instance in a virtual machine, MariaDB in a different virtual maching and MSSQL-server with a Docker container based on mcr.microsoft.com/mssql/server, which is MSSQL-2017. I can share the Dockerfile for that. We can forget about Firebird. It can't handle concurrency well because its concurrency model doesn't follow the SQL Standard. I will eventually drop that again anyway. These are of course only functional tests. None of those database installations can handle any meaningful load.

Once files are moved, then the entire code will be adjusted. * The code will be formatted to Eclipse syntax format. * The imports will be organized. * Packages will be used for all classes.

Those changes will definitely help students to find their way around the code easier.

* Put a FIXME or TODO, in each part of the code where an extension should be done for other RDBMS.

The really RDBMS specific stuff for stored procedures is in separate files now in the ./application directory. The AppGeneric driver, which implements all transactions inside the application with preparedStatment()s, there is exactly one single query left that is RDBMS vendor specific. And that is the STOCK_LEVEL query because Oracle doesn't like AS <ident> after a subselect while other databases either tolerate or require it. Which means that adopting support for a new vendor requires writing a new ./application/App<vendor>StoredProc.java (if stored procedure support is being implemented) and touching that one spot in the AppGeneric.java file.

* Use Wiki, to explain things about this project. How to use the python script, generation of the diagrams with r, etc.

Don't start on documenting the R stuff. I intend to replace the usage of R with matplotlib and basically rewrite the entire report generation as well as the result data capture from scratch. But I agree on using the Wiki.

o In the wiki, there could be a page per RDBMS to explain the details and points of performance improvement. * Create javadoc headers, in order to be capable of generating a documentation of the code. * Documentation about the parameters that receives the application. * Extra parameters, like: o schema, which is useful for databases like MySQL that do not use schemas. Or for other databases which uses default schema. This do not force to use benchmarksql schema. o Statement terminator, when ; is not available. This could use another one like 'GO' or '@', or even EOL.

The current master implementation does not use schemas. Everything in PostgreSQL for example is int the "public" schema. All tables have a bmsql_ prefix. It also doesn't use GO or the like since the SQL files for building and dropping the tables/procedures are processed by a Java utility program.

* Use PMD, findbugs and checkstyle for better code. * Use of Travis-CI to do Continuous Integration. * Generation of a GitHub web page, based on the files under doc. * Include support to other RDBMS, starting with Db2, and then extending to others. Finally, I have theses questions, I need your answer to proceed: * It is a good practice to have packages for all classes. I propose to use the following package name base, that easily reference the class to the hosting facility of the code: com.github.pgsql-io.benchmarksql

BenchmarkSQL has moved the main repository before. It used to be on bitbucket and then moved here. I don't know how long term stable the pgsql-io part the pgsql-io part of the above will be. Other than that I am for it.

Under this package, all classes will be placed under different sub packages (jtpcc, loader, etc.) * Which minimum java version BenchmarkSQL should support? o Java 1.7 is not longer maintained, but very popular. o Java 8 is still maintained and very popular.

I'm still using 8, but should test against 11 once I get a few more resources (mostly time).

o Any older one? o Any newer, like java 9 with Lambda expressions?

I wouldn't want to require a newer one by using features, that aren't in Java 8. Best Regards, Jan

…

-- Jan Wieck Principle Database Engineer Amazon Web Services

[luss]

in my (limited) experience, 11 is almost totally upwardly compatible with 8, but... there are a few deprecations that cause various compile time warnings. To the degree that benchmark sql ships with class files already made, we'd do that with JDK8 so the warnings wouldn't be a short term issue until version of JDK after 11.

…

--Luss

On Thu, Apr 1, 2021 at 12:51 PM Jan Wieck ***@***.***> wrote: I suggest you make yourself familiar with the current master branch of benchmarksql. There have been substantial changes to some of what you are touching below. More comments inline. On 4/1/21 11:55 AM, Andres Gomez Casanova wrote: > Hi guys, > > The code modernizations that I propose do not change the current > functionality, and this won't correct or add new functionality. It just > will use new frameworks that eases the deployment. The changes I propose > will be done in parts, in order to have a tracking of the evolution; not > just a big change. > The important thing to take into account with this refactoring is that > the firsts "pull request" will have a lot of visual changes: > > * The files will be moved to other directories to comply with Maven > structure. > * The shell scripts will be modified to support Maven structure in the > |target| directory instead of the |run| directory. New directory structure is fine. > * Libraries like drivers will be automatically downloaded and included > in the target directory by Maven. This I am very interested in. I was looking at the Maven Repository at https://repo1.maven.org/maven2 earlier and noticed that the Oracle and Microsoft JDBC drivers are there. Oracle even points to that repository on their download page. Which means that Oracle has changed their policy and no longer requires license click through for download. Bravo! I still don't think we should ship them or add copies of them to our repository, but if they are automatically downloaded at build time and if a user has an easy way to get them into a local lib directory to run a pre-built Docker container, that would be perfect. > * All documentation in one directory, separating it from code or scripts. > > I will adjust the scripts to work with Postgres; currently I cannot test > against Oracle. We definitely need to make sure that the changes do not interfere with the support for Oracle, MariaDB or MSSQL-server. Currently I am testing Oracle against an Oracle-11g instance in a virtual machine, MariaDB in a different virtual maching and MSSQL-server with a Docker container based on mcr.microsoft.com/mssql/server, which is MSSQL-2017. I can share the Dockerfile for that. We can forget about Firebird. It can't handle concurrency well because its concurrency model doesn't follow the SQL Standard. I will eventually drop that again anyway. These are of course only functional tests. None of those database installations can handle any meaningful load. > > Once files are moved, then the entire code will be adjusted. > > * The code will be formatted to Eclipse syntax format. > * The imports will be organized. > * Packages will be used for all classes. Those changes will definitely help students to find their way around the code easier. > * Put a FIXME or TODO, in each part of the code where an extension > should be done for other RDBMS. The really RDBMS specific stuff for stored procedures is in separate files now in the ./application directory. The AppGeneric driver, which implements all transactions inside the application with preparedStatment()s, there is exactly one single query left that is RDBMS vendor specific. And that is the STOCK_LEVEL query because Oracle doesn't like AS <ident> after a subselect while other databases either tolerate or require it. Which means that adopting support for a new vendor requires writing a new ./application/App<vendor>StoredProc.java (if stored procedure support is being implemented) and touching that one spot in the AppGeneric.java file. > * Use Wiki, to explain things about this project. How to use the > python script, generation of the diagrams with r, etc. Don't start on documenting the R stuff. I intend to replace the usage of R with matplotlib and basically rewrite the entire report generation as well as the result data capture from scratch. But I agree on using the Wiki. > o In the wiki, there could be a page per RDBMS to explain the > details and points of performance improvement. > * Create javadoc headers, in order to be capable of generating a > documentation of the code. > * Documentation about the parameters that receives the application. > * Extra parameters, like: > o schema, which is useful for databases like MySQL that do not use > schemas. Or for other databases which uses default schema. This > do not force to use benchmarksql schema. > o Statement terminator, when ; is not available. This could use > another one like 'GO' or '@', or even EOL. The current master implementation does not use schemas. Everything in PostgreSQL for example is int the "public" schema. All tables have a bmsql_ prefix. It also doesn't use GO or the like since the SQL files for building and dropping the tables/procedures are processed by a Java utility program. > * Use PMD, findbugs and checkstyle for better code. > * Use of Travis-CI to do Continuous Integration. > * Generation of a GitHub web page, based on the files under doc. > * Include support to other RDBMS, starting with Db2, and then > extending to others. > > Finally, I have theses questions, I need your answer to proceed: > > * > > It is a good practice to have packages for all classes. I propose to > use the following package name base, that easily reference the class > to the hosting facility of the code: > > com.github.pgsql-io.benchmarksql BenchmarkSQL has moved the main repository before. It used to be on bitbucket and then moved here. I don't know how long term stable the pgsql-io part the pgsql-io part of the above will be. Other than that I am for it. > > Under this package, all classes will be placed under different sub > packages (jtpcc, loader, etc.) > > * Which minimum java version BenchmarkSQL should support? > o Java 1.7 is not longer maintained, but very popular. > o Java 8 is still maintained and very popular. I'm still using 8, but should test against 11 once I get a few more resources (mostly time). > o Any older one? > o Any newer, like java 9 with Lambda expressions? I wouldn't want to require a newer one by using features, that aren't in Java 8. Best Regards, Jan -- Jan Wieck Principle Database Engineer Amazon Web Services — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMWOHSPGZKPARESLUQKKETTGSQCXANCNFSM4Z6LRPBQ> .

[wieck]

On 4/1/21 1:52 PM, Denis Lussier wrote: in my (limited) experience, 11 is almost totally upwardly compatible with 8, but... there are a few deprecations that cause various compile time warnings. To the degree that benchmark sql ships with class files already made, we'd do that with JDK8 so the warnings wouldn't be a short term issue until version of JDK after 11.

I just ran into another problem with 8 while trying to use the ojdbc10 driver: Exception in thread "main" java.lang.UnsupportedClassVersionError: oracle/jdbc/OracleDriver has been compiled by a more recent version of the Java Runtime (class file version 54.0), this version of the Java Runtime only recognizes class file versions up to 52.0 So I guess we will have to move to something newer than java-1.8. Regards, Jan

…

--Luss On Thu, Apr 1, 2021 at 12:51 PM Jan Wieck ***@***.***> wrote: > I suggest you make yourself familiar with the current master branch of > benchmarksql. There have been substantial changes to some of what you > are touching below. More comments inline. > > On 4/1/21 11:55 AM, Andres Gomez Casanova wrote: > > Hi guys, > > > > The code modernizations that I propose do not change the current > > functionality, and this won't correct or add new functionality. It just > > will use new frameworks that eases the deployment. The changes I propose > > will be done in parts, in order to have a tracking of the evolution; not > > just a big change. > > The important thing to take into account with this refactoring is that > > the firsts "pull request" will have a lot of visual changes: > > > > * The files will be moved to other directories to comply with Maven > > structure. > > * The shell scripts will be modified to support Maven structure in the > > |target| directory instead of the |run| directory. > > New directory structure is fine. > > > * Libraries like drivers will be automatically downloaded and included > > in the target directory by Maven. > > This I am very interested in. I was looking at the Maven Repository at > https://repo1.maven.org/maven2 earlier and noticed that the Oracle and > Microsoft JDBC drivers are there. Oracle even points to that repository > on their download page. Which means that Oracle has changed their policy > and no longer requires license click through for download. Bravo! > > I still don't think we should ship them or add copies of them to our > repository, but if they are automatically downloaded at build time and > if a user has an easy way to get them into a local lib directory to run > a pre-built Docker container, that would be perfect. > > > * All documentation in one directory, separating it from code or scripts. > > > > I will adjust the scripts to work with Postgres; currently I cannot test > > against Oracle. > > We definitely need to make sure that the changes do not interfere with > the support for Oracle, MariaDB or MSSQL-server. Currently I am testing > Oracle against an Oracle-11g instance in a virtual machine, MariaDB in a > different virtual maching and MSSQL-server with a Docker container based > on mcr.microsoft.com/mssql/server, which is MSSQL-2017. I can share the > Dockerfile for that. > > We can forget about Firebird. It can't handle concurrency well because > its concurrency model doesn't follow the SQL Standard. I will eventually > drop that again anyway. > > These are of course only functional tests. None of those database > installations can handle any meaningful load. > > > > > Once files are moved, then the entire code will be adjusted. > > > > * The code will be formatted to Eclipse syntax format. > > * The imports will be organized. > > * Packages will be used for all classes. > > Those changes will definitely help students to find their way around the > code easier. > > > * Put a FIXME or TODO, in each part of the code where an extension > > should be done for other RDBMS. > > The really RDBMS specific stuff for stored procedures is in separate > files now in the ./application directory. > > The AppGeneric driver, which implements all transactions inside the > application with preparedStatment()s, there is exactly one single query > left that is RDBMS vendor specific. And that is the STOCK_LEVEL query > because Oracle doesn't like AS <ident> after a subselect while other > databases either tolerate or require it. > > Which means that adopting support for a new vendor requires writing a > new ./application/App<vendor>StoredProc.java (if stored procedure > support is being implemented) and touching that one spot in the > AppGeneric.java file. > > > * Use Wiki, to explain things about this project. How to use the > > python script, generation of the diagrams with r, etc. > > Don't start on documenting the R stuff. I intend to replace the usage of > R with matplotlib and basically rewrite the entire report generation as > well as the result data capture from scratch. > > But I agree on using the Wiki. > > > o In the wiki, there could be a page per RDBMS to explain the > > details and points of performance improvement. > > * Create javadoc headers, in order to be capable of generating a > > documentation of the code. > > * Documentation about the parameters that receives the application. > > * Extra parameters, like: > > o schema, which is useful for databases like MySQL that do not use > > schemas. Or for other databases which uses default schema. This > > do not force to use benchmarksql schema. > > o Statement terminator, when ; is not available. This could use > > another one like 'GO' or '@', or even EOL. > > The current master implementation does not use schemas. Everything in > PostgreSQL for example is int the "public" schema. All tables have a > bmsql_ prefix. It also doesn't use GO or the like since the SQL files > for building and dropping the tables/procedures are processed by a Java > utility program. > > > * Use PMD, findbugs and checkstyle for better code. > > * Use of Travis-CI to do Continuous Integration. > > * Generation of a GitHub web page, based on the files under doc. > > * Include support to other RDBMS, starting with Db2, and then > > extending to others. > > > > Finally, I have theses questions, I need your answer to proceed: > > > > * > > > > It is a good practice to have packages for all classes. I propose to > > use the following package name base, that easily reference the class > > to the hosting facility of the code: > > > > com.github.pgsql-io.benchmarksql > > BenchmarkSQL has moved the main repository before. It used to be on > bitbucket and then moved here. I don't know how long term stable the > pgsql-io part the pgsql-io part of the above will be. Other than that I > am for it. > > > > > Under this package, all classes will be placed under different sub > > packages (jtpcc, loader, etc.) > > > > * Which minimum java version BenchmarkSQL should support? > > o Java 1.7 is not longer maintained, but very popular. > > o Java 8 is still maintained and very popular. > > I'm still using 8, but should test against 11 once I get a few more > resources (mostly time). > > > o Any older one? > > o Any newer, like java 9 with Lambda expressions? > > I wouldn't want to require a newer one by using features, that aren't in > Java 8. > > > Best Regards, Jan > > -- > Jan Wieck > Principle Database Engineer > Amazon Web Services > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#3 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAMWOHSPGZKPARESLUQKKETTGSQCXANCNFSM4Z6LRPBQ> > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACYRMTGTSPPVCGAWHSZ3QTTGSXGFANCNFSM4Z6LRPBQ>.

-- Jan Wieck Principle Database Engineer Amazon Web Services

[luss]

11 it is then. It's been out for a couple years. If you look at the versions of JDK available in newer os versions (such as centos-8 and ubuntu-20.04), it is both OpenJDK 8 and OpenJDK 11. Seems like every three years Oracle and the community are supporting LTS releases.

…

On Thu, Apr 1, 2021 at 2:10 PM Jan Wieck ***@***.***> wrote: On 4/1/21 1:52 PM, Denis Lussier wrote: > in my (limited) experience, 11 is almost totally upwardly compatible with > 8, but... there are a few deprecations that cause various compile time > warnings. > To the degree that benchmark sql ships with class files already made, we'd > do that with JDK8 so the warnings wouldn't be a short term issue until > version of JDK after 11. I just ran into another problem with 8 while trying to use the ojdbc10 driver: Exception in thread "main" java.lang.UnsupportedClassVersionError: oracle/jdbc/OracleDriver has been compiled by a more recent version of the Java Runtime (class file version 54.0), this version of the Java Runtime only recognizes class file versions up to 52.0 So I guess we will have to move to something newer than java-1.8. Regards, Jan > > --Luss > > On Thu, Apr 1, 2021 at 12:51 PM Jan Wieck ***@***.***> wrote: > > > I suggest you make yourself familiar with the current master branch of > > benchmarksql. There have been substantial changes to some of what you > > are touching below. More comments inline. > > > > On 4/1/21 11:55 AM, Andres Gomez Casanova wrote: > > > Hi guys, > > > > > > The code modernizations that I propose do not change the current > > > functionality, and this won't correct or add new functionality. It just > > > will use new frameworks that eases the deployment. The changes I > propose > > > will be done in parts, in order to have a tracking of the > evolution; not > > > just a big change. > > > The important thing to take into account with this refactoring is that > > > the firsts "pull request" will have a lot of visual changes: > > > > > > * The files will be moved to other directories to comply with Maven > > > structure. > > > * The shell scripts will be modified to support Maven structure in the > > > |target| directory instead of the |run| directory. > > > > New directory structure is fine. > > > > > * Libraries like drivers will be automatically downloaded and included > > > in the target directory by Maven. > > > > This I am very interested in. I was looking at the Maven Repository at > > https://repo1.maven.org/maven2 earlier and noticed that the Oracle and > > Microsoft JDBC drivers are there. Oracle even points to that repository > > on their download page. Which means that Oracle has changed their policy > > and no longer requires license click through for download. Bravo! > > > > I still don't think we should ship them or add copies of them to our > > repository, but if they are automatically downloaded at build time and > > if a user has an easy way to get them into a local lib directory to run > > a pre-built Docker container, that would be perfect. > > > > > * All documentation in one directory, separating it from code or > scripts. > > > > > > I will adjust the scripts to work with Postgres; currently I cannot > test > > > against Oracle. > > > > We definitely need to make sure that the changes do not interfere with > > the support for Oracle, MariaDB or MSSQL-server. Currently I am testing > > Oracle against an Oracle-11g instance in a virtual machine, MariaDB in a > > different virtual maching and MSSQL-server with a Docker container based > > on mcr.microsoft.com/mssql/server, which is MSSQL-2017. I can share the > > Dockerfile for that. > > > > We can forget about Firebird. It can't handle concurrency well because > > its concurrency model doesn't follow the SQL Standard. I will eventually > > drop that again anyway. > > > > These are of course only functional tests. None of those database > > installations can handle any meaningful load. > > > > > > > > Once files are moved, then the entire code will be adjusted. > > > > > > * The code will be formatted to Eclipse syntax format. > > > * The imports will be organized. > > > * Packages will be used for all classes. > > > > Those changes will definitely help students to find their way around the > > code easier. > > > > > * Put a FIXME or TODO, in each part of the code where an extension > > > should be done for other RDBMS. > > > > The really RDBMS specific stuff for stored procedures is in separate > > files now in the ./application directory. > > > > The AppGeneric driver, which implements all transactions inside the > > application with preparedStatment()s, there is exactly one single query > > left that is RDBMS vendor specific. And that is the STOCK_LEVEL query > > because Oracle doesn't like AS <ident> after a subselect while other > > databases either tolerate or require it. > > > > Which means that adopting support for a new vendor requires writing a > > new ./application/App<vendor>StoredProc.java (if stored procedure > > support is being implemented) and touching that one spot in the > > AppGeneric.java file. > > > > > * Use Wiki, to explain things about this project. How to use the > > > python script, generation of the diagrams with r, etc. > > > > Don't start on documenting the R stuff. I intend to replace the usage of > > R with matplotlib and basically rewrite the entire report generation as > > well as the result data capture from scratch. > > > > But I agree on using the Wiki. > > > > > o In the wiki, there could be a page per RDBMS to explain the > > > details and points of performance improvement. > > > * Create javadoc headers, in order to be capable of generating a > > > documentation of the code. > > > * Documentation about the parameters that receives the application. > > > * Extra parameters, like: > > > o schema, which is useful for databases like MySQL that do not use > > > schemas. Or for other databases which uses default schema. This > > > do not force to use benchmarksql schema. > > > o Statement terminator, when ; is not available. This could use > > > another one like 'GO' or '@', or even EOL. > > > > The current master implementation does not use schemas. Everything in > > PostgreSQL for example is int the "public" schema. All tables have a > > bmsql_ prefix. It also doesn't use GO or the like since the SQL files > > for building and dropping the tables/procedures are processed by a Java > > utility program. > > > > > * Use PMD, findbugs and checkstyle for better code. > > > * Use of Travis-CI to do Continuous Integration. > > > * Generation of a GitHub web page, based on the files under doc. > > > * Include support to other RDBMS, starting with Db2, and then > > > extending to others. > > > > > > Finally, I have theses questions, I need your answer to proceed: > > > > > > * > > > > > > It is a good practice to have packages for all classes. I propose to > > > use the following package name base, that easily reference the class > > > to the hosting facility of the code: > > > > > > com.github.pgsql-io.benchmarksql > > > > BenchmarkSQL has moved the main repository before. It used to be on > > bitbucket and then moved here. I don't know how long term stable the > > pgsql-io part the pgsql-io part of the above will be. Other than that I > > am for it. > > > > > > > > Under this package, all classes will be placed under different sub > > > packages (jtpcc, loader, etc.) > > > > > > * Which minimum java version BenchmarkSQL should support? > > > o Java 1.7 is not longer maintained, but very popular. > > > o Java 8 is still maintained and very popular. > > > > I'm still using 8, but should test against 11 once I get a few more > > resources (mostly time). > > > > > o Any older one? > > > o Any newer, like java 9 with Lambda expressions? > > > > I wouldn't want to require a newer one by using features, that aren't in > > Java 8. > > > > > > Best Regards, Jan > > > > -- > > Jan Wieck > > Principle Database Engineer > > Amazon Web Services > > > > — > > You are receiving this because you commented. > > Reply to this email directly, view it on GitHub > > > < #3 (comment)>, > > or unsubscribe > > > < https://github.com/notifications/unsubscribe-auth/AAMWOHSPGZKPARESLUQKKETTGSQCXANCNFSM4Z6LRPBQ > > > . > > > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > < #3 (comment)>, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AACYRMTGTSPPVCGAWHSZ3QTTGSXGFANCNFSM4Z6LRPBQ >. > -- Jan Wieck Principle Database Engineer Amazon Web Services — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMWOHT2IDBJ6ZD22BE65O3TGSZLFANCNFSM4Z6LRPBQ> .

[wieck]

On 4/1/21 2:44 PM, Denis Lussier wrote: 11 it is then. It's been out for a couple years. If you look at the versions of JDK available in newer os versions (such as centos-8 and ubuntu-20.04), it is both OpenJDK 8 and OpenJDK 11. Seems like every three years Oracle and the community are supporting LTS releases.

Unfortunately Oracle is retarded. Even with the newest drivers we have: "Oracle JDBC does not support the JDBC 4.0 method createArrayOf method of java.sql.Connection interface. This method only allows anonymous array types, while all Oracle array types are named. Use the Oracle specific method oracle.jdbc.OracleConnection.createARRAY instead." So it seems impossible to compile Oracle Stored Procedure support without having the Oracle driver present. It might be possible to build with Oracle support, not include the Oracle driver in the Docker image, but then put it back into place at runtime. Regards, Jan

…

On Thu, Apr 1, 2021 at 2:10 PM Jan Wieck ***@***.***> wrote: > On 4/1/21 1:52 PM, Denis Lussier wrote: > > in my (limited) experience, 11 is almost totally upwardly compatible with > > 8, but... there are a few deprecations that cause various compile time > > warnings. > > To the degree that benchmark sql ships with class files already made, > we'd > > do that with JDK8 so the warnings wouldn't be a short term issue until > > version of JDK after 11. > > I just ran into another problem with 8 while trying to use the ojdbc10 > driver: > > Exception in thread "main" java.lang.UnsupportedClassVersionError: > oracle/jdbc/OracleDriver has been compiled by a more recent version of > the Java Runtime (class file version 54.0), this version of the Java > Runtime only recognizes class file versions up to 52.0 > > So I guess we will have to move to something newer than java-1.8. > > > Regards, Jan > > > > > > --Luss > > > > On Thu, Apr 1, 2021 at 12:51 PM Jan Wieck ***@***.***> wrote: > > > > > I suggest you make yourself familiar with the current master branch of > > > benchmarksql. There have been substantial changes to some of what you > > > are touching below. More comments inline. > > > > > > On 4/1/21 11:55 AM, Andres Gomez Casanova wrote: > > > > Hi guys, > > > > > > > > The code modernizations that I propose do not change the current > > > > functionality, and this won't correct or add new functionality. It > just > > > > will use new frameworks that eases the deployment. The changes I > > propose > > > > will be done in parts, in order to have a tracking of the > > evolution; not > > > > just a big change. > > > > The important thing to take into account with this refactoring is > that > > > > the firsts "pull request" will have a lot of visual changes: > > > > > > > > * The files will be moved to other directories to comply with Maven > > > > structure. > > > > * The shell scripts will be modified to support Maven structure in > the > > > > |target| directory instead of the |run| directory. > > > > > > New directory structure is fine. > > > > > > > * Libraries like drivers will be automatically downloaded and > included > > > > in the target directory by Maven. > > > > > > This I am very interested in. I was looking at the Maven Repository at > > > https://repo1.maven.org/maven2 earlier and noticed that the Oracle and > > > Microsoft JDBC drivers are there. Oracle even points to that repository > > > on their download page. Which means that Oracle has changed their > policy > > > and no longer requires license click through for download. Bravo! > > > > > > I still don't think we should ship them or add copies of them to our > > > repository, but if they are automatically downloaded at build time and > > > if a user has an easy way to get them into a local lib directory to run > > > a pre-built Docker container, that would be perfect. > > > > > > > * All documentation in one directory, separating it from code or > > scripts. > > > > > > > > I will adjust the scripts to work with Postgres; currently I cannot > > test > > > > against Oracle. > > > > > > We definitely need to make sure that the changes do not interfere with > > > the support for Oracle, MariaDB or MSSQL-server. Currently I am testing > > > Oracle against an Oracle-11g instance in a virtual machine, MariaDB in > a > > > different virtual maching and MSSQL-server with a Docker container > based > > > on mcr.microsoft.com/mssql/server, which is MSSQL-2017. I can share > the > > > Dockerfile for that. > > > > > > We can forget about Firebird. It can't handle concurrency well because > > > its concurrency model doesn't follow the SQL Standard. I will > eventually > > > drop that again anyway. > > > > > > These are of course only functional tests. None of those database > > > installations can handle any meaningful load. > > > > > > > > > > > Once files are moved, then the entire code will be adjusted. > > > > > > > > * The code will be formatted to Eclipse syntax format. > > > > * The imports will be organized. > > > > * Packages will be used for all classes. > > > > > > Those changes will definitely help students to find their way around > the > > > code easier. > > > > > > > * Put a FIXME or TODO, in each part of the code where an extension > > > > should be done for other RDBMS. > > > > > > The really RDBMS specific stuff for stored procedures is in separate > > > files now in the ./application directory. > > > > > > The AppGeneric driver, which implements all transactions inside the > > > application with preparedStatment()s, there is exactly one single query > > > left that is RDBMS vendor specific. And that is the STOCK_LEVEL query > > > because Oracle doesn't like AS <ident> after a subselect while other > > > databases either tolerate or require it. > > > > > > Which means that adopting support for a new vendor requires writing a > > > new ./application/App<vendor>StoredProc.java (if stored procedure > > > support is being implemented) and touching that one spot in the > > > AppGeneric.java file. > > > > > > > * Use Wiki, to explain things about this project. How to use the > > > > python script, generation of the diagrams with r, etc. > > > > > > Don't start on documenting the R stuff. I intend to replace the usage > of > > > R with matplotlib and basically rewrite the entire report generation as > > > well as the result data capture from scratch. > > > > > > But I agree on using the Wiki. > > > > > > > o In the wiki, there could be a page per RDBMS to explain the > > > > details and points of performance improvement. > > > > * Create javadoc headers, in order to be capable of generating a > > > > documentation of the code. > > > > * Documentation about the parameters that receives the application. > > > > * Extra parameters, like: > > > > o schema, which is useful for databases like MySQL that do not use > > > > schemas. Or for other databases which uses default schema. This > > > > do not force to use benchmarksql schema. > > > > o Statement terminator, when ; is not available. This could use > > > > another one like 'GO' or '@', or even EOL. > > > > > > The current master implementation does not use schemas. Everything in > > > PostgreSQL for example is int the "public" schema. All tables have a > > > bmsql_ prefix. It also doesn't use GO or the like since the SQL files > > > for building and dropping the tables/procedures are processed by a Java > > > utility program. > > > > > > > * Use PMD, findbugs and checkstyle for better code. > > > > * Use of Travis-CI to do Continuous Integration. > > > > * Generation of a GitHub web page, based on the files under doc. > > > > * Include support to other RDBMS, starting with Db2, and then > > > > extending to others. > > > > > > > > Finally, I have theses questions, I need your answer to proceed: > > > > > > > > * > > > > > > > > It is a good practice to have packages for all classes. I propose to > > > > use the following package name base, that easily reference the class > > > > to the hosting facility of the code: > > > > > > > > com.github.pgsql-io.benchmarksql > > > > > > BenchmarkSQL has moved the main repository before. It used to be on > > > bitbucket and then moved here. I don't know how long term stable the > > > pgsql-io part the pgsql-io part of the above will be. Other than that I > > > am for it. > > > > > > > > > > > Under this package, all classes will be placed under different sub > > > > packages (jtpcc, loader, etc.) > > > > > > > > * Which minimum java version BenchmarkSQL should support? > > > > o Java 1.7 is not longer maintained, but very popular. > > > > o Java 8 is still maintained and very popular. > > > > > > I'm still using 8, but should test against 11 once I get a few more > > > resources (mostly time). > > > > > > > o Any older one? > > > > o Any newer, like java 9 with Lambda expressions? > > > > > > I wouldn't want to require a newer one by using features, that aren't > in > > > Java 8. > > > > > > > > > Best Regards, Jan > > > > > > -- > > > Jan Wieck > > > Principle Database Engineer > > > Amazon Web Services > > > > > > — > > > You are receiving this because you commented. > > > Reply to this email directly, view it on GitHub > > > > > < > #3 (comment)>, > > > or unsubscribe > > > > > < > https://github.com/notifications/unsubscribe-auth/AAMWOHSPGZKPARESLUQKKETTGSQCXANCNFSM4Z6LRPBQ > > > > > . > > > > > > > — > > You are receiving this because you commented. > > Reply to this email directly, view it on GitHub > > < > #3 (comment)>, > > > or unsubscribe > > < > https://github.com/notifications/unsubscribe-auth/AACYRMTGTSPPVCGAWHSZ3QTTGSXGFANCNFSM4Z6LRPBQ > >. > > > > > -- > Jan Wieck > Principle Database Engineer > Amazon Web Services > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#3 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAMWOHT2IDBJ6ZD22BE65O3TGSZLFANCNFSM4Z6LRPBQ> > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACYRMS6YUSBNFEIQ46ZK53TGS5HVANCNFSM4Z6LRPBQ>.

-- Jan Wieck Principle Database Engineer Amazon Web Services