Fixed to work with newest version of Django. Fixes for viewing and downloading large reports.

[agronick]

I've been using qbe. I made a bunch of modifications to it - making it work outside the admin for less experienced users. Along the way I found a few bugs and things needed to be modified for the newer versions of Django. I pulled out some of the more important changes to integrate upstream. The version I'm committing works on Python 2 with Django 1.11.3. It doesn't add any more features. It allows reports of any size to be displayed or downloaded without overflowing the available memory. They also download several hundred times faster. I had to remove the ability to display links in the report in order to fix this.

I didn't commit the changes needed for Python 3. You should probably run 2to3 to get the codebase to work with Python3. I did that myself and it didn't take more than an hour. The changes I'm committing are only tested with Python 2.

Heres the git commit details:
Using StringIO to write one line at a time before flushing it is very slow. Reports download at about 5kbps. Also, previously the reports were created in memory and large reports would cause the server to run out of memory. This change causes the results to be written to the response while they are being read from the database. It never runs out of memory and runs at 600kbps.
The cursor is used on the HTML view as well. The same problem can present itself if the user selects to view all records. The raw cursor is read instead of a list of results.

[versae]

Wow. Thank you so much! It's a lot of changes. I'll review and find a way to integrate all the changes.

[agronick]

Looks like I left a pdb statement in there by accident. I have a lot of other changes - some of them are specific to my app so they're kind of hard to pull out.

Heres my fill fields event code. It shows different criteria filters depending on the type of field.

• For a join it hides the criteria box and just puts join to
• For a date it shows a datepicker -- needs to be added or switched to something else
• For a time it shows a timepicker
• For a boolean it shows true or false dropdown

        /**
         * Event triggered when the SELECT tag for fill fields is changed
         */
        qbe.Core.fillFieldsEvent = function() {
            var field, splits, prefix, css, cssSplit, inputs, input, domTo, appModel, appModelSplits, fields, primary, target, targetRel, targetModel, targetStrings, targetString, relations;
            field = $(this).val();
            splits = $(this).attr("id").split("-");
            prefix = splits.splice(0, splits.length - 1).join("-");
            css = $(this).attr("class");
            cssSplit = css.split("enable:");
            inputs = cssSplit[cssSplit.length - 1].split(",");
            for (var i = 0; i < inputs.length; i++) {
                input = inputs[i];
                domTo = prefix + "-" + input;
                if (field) {
                    $("#" + domTo).removeAttr("disabled");
                } else {
                    $("#" + domTo).attr("disabled", "disabled");
                    $("#" + domTo).val("");
                }
                if ($("#" + domTo).is(':input')) {
                    appModel = $("#" + prefix + "-model").val();
                    if (appModel) {
                        appModelSplits = appModel.split(".");
                        fields = qbe.Models[appModelSplits[0]][appModelSplits[1]].fields;
                        if (field in fields && fields[field].target && !_loadingData) {
                            target = fields[field].target;
                            if (target.through) {
                                $(this).parent().parent().children("td:last").children("a").click();
                                targetModel = qbe.Models[target.through.name][target.through.model];
                                targetsString = [];
                                relations = targetModel.relations;
                                for (var r = 0; r < targetModel.relations.length; r++) {
                                    targetRel = targetModel.relations[r];
                                    targetString = target.through.name + "." + target.through.model + "." + targetRel.source;
                                    targetsString.push(targetString);
                                }
                                qbe.Core.addRelationsFrom(targetsString.join("-"));
                            } else {
                                targetString = target.name + "." + target.model + "." + target.field;
                                $("#" + domTo).val(targetString);
                                $("#" + domTo).prev().val("join");
                                qbe.Core.addRelationsFrom(targetString);
                            }
                        } else {
                            $("#" + domTo).val("");
                        }
                    }
                }
            }


            var field_type = '';
            try {
                field_type = qbe.Models[appModelSplits[0]][appModelSplits[1]].fields[field].type.toLowerCase();
            } catch (e) {}

            if (typeof qbe.pickers === 'undefined') {
                qbe.pickers = [];
            }

            var filter_select = $('#' + domTo).prev();
            filter_select.parent().children().show();
            filter_select.find('option[value=join]').remove();
            filter_select.prev('.join_lbl').remove();
            filter_select.parents('tr:first').children('td:first').find('label').show();
            $('#' + prefix + '-sort').show();


            // First destroy any special field changes
            // Destroy the date picker
            if (typeof qbe.pickers[domTo] !== 'undefined' && qbe.pickers[domTo] !== null)
                qbe.pickers[domTo].destroy();
            qbe.pickers[domTo] = null;

            filter_select.parents('tr:first').find('td:first label').show();

            // Destroy the boolean dropdown
            var criteria_field = $('#' + domTo);

            // Remove timefield
            criteria_field.filter('.time_input').removeClass('time_input').unbind('change').next('span').remove();

            // Remove boolfield
            criteria_field.prev('.bool_lbl').remove();
            if (criteria_field.hasClass('bool_select')) {
                criteria_field.replaceWith(
                    $('<input>').attr({
                        'type': 'text',
                        'name': criteria_field.attr('name'),
                        'id': criteria_field.attr('id')
                    })
                );
            }

            if (field_type === 'foreignkey' || field_type == "onetoonefield") {
                var filter_select = $('#' + domTo).prev();
                if (filter_select.has('option[value=join]').length == 0) {
                    filter_select.append($('<option>').attr('value', 'join').text('joins to'));
                }
                filter_select.val('join').next('input').hide();
                var jointo_model = filter_select.parents('tr:first').find('.qbeFillFields > :selected').text();
                filter_select.parents('tr:first').find('td:first input').prop('checked', null).next().hide();
                filter_select.children().not('[value=None], [value=join]').hide();
                filter_select.children('[value=join]').append(' ' + jointo_model);
                filter_select.children('[value=None]').html('has no ' + jointo_model);

            } else if (field_type.indexOf('date') !== -1) {
                var picker = new Pikaday({
                    field: document.getElementById(domTo),
                    format: 'YYYY-MM-DD'
                });
                qbe.pickers[domTo] = picker;

            // Or add the boolean dropdown
            } else if(field_type.indexOf('bool') !== -1) {
                var text_input = $('#' + domTo);
                filter_select.hide();
                text_input.replaceWith(
                    $('<select>').append(
                        $('<option>').attr('value', '').text('is True or False')
                    ).append(
                        $('<option>').attr('value', '1').text('is True')
                    ).append(
                        $('<option>').attr('value', '0').text('is False')
                    ).attr({
                        'name': text_input.attr('name'),
                        'id': text_input.attr('id')
                    }).change(function(){
                        filter_select.val($(this).val().length ? 'exact' : '');
                    }).change().addClass('bool_select').before(
                        $('<strong>').text('is equal to').addClass('bool_lbl')
                    )
                )
            } else if(field_type == 'timefield'){
                var text_input = $('#' + domTo).change(function(){
                    var parts = $(this).val().split(':');
                    var text_input = $(this);
                    ['hour', 'minute'].forEach(function(field, i){
                        if (parts[i] != undefined && !isNaN(parts[i]))
                            text_input.next().children('.' + field + '_select').val(parseInt(parts[i]));
                    });

                }).addClass('time_input').hide();

                var wrap = $('<span>');

                var hours = $('<select>').addClass('hour_select');
                for (var i = 0; i < 24; i++)
                    hours.append($('<option>').attr('value', i).html(('0' + i).slice(-2)));

                var minutes = $('<select>').addClass('minute_select');
                for (var i = 0; i < 60; i++)
                    minutes.append($('<option>').attr('value', i).html(('0' + i).slice(-2)));

                wrap.append(hours).append($('<strong>').html(' : ')).append(minutes).change(function(){
                    $(this).prev().val(
                        hours.val() + ':' + minutes.val() + ':00'
                    );
                });
                text_input.after(wrap);
            }
            filter_select.change();
        };

[agronick]

I uploaded my whole build directory. Its tightly integrated so it won't work as is but theres some good snippits worth pulling out.

agronick@05913f7?diff=unified

• In qbe.diagram.js there is code for making a good effort that the tables in the diagram don't overlap.
• forms.py checks that everything is joined so you don't have queries where everything is joined to every other thing.
• In operator and null operator

[versae]

Wow, this is some pretty amazing work. Not sure when I'll be able to review it all and integrate it, but thank you so much!

…

On Thu, Jan 11, 2018 at 2:28 PM, Kyle Agronick ***@***.***> wrote: I uploaded my whole build directory. Its tightly integrated so it won't work as is but theres some good snippits worth pulling out. ***@***.***?diff=unified <agronick@05913f7?diff=unified> - In qbe.diagram.js there is code for making a good effort that the tables in the diagram don't overlap. - forms.py checks that everything is joined so you don't have queries where everything is joined to every other thing. - In operator and null operator — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#54 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKl4WnKSbunAmCSmk5wVXxaCJDquuRdks5tJosAgaJpZM4OUgX8> .

-- Javier de la Rosa http://versae.es