No internet connection
  1. Home
  2. How to

Simulate mouse drag from relative UI position

By Kitch Membery @Kitch2020-04-18 22:33:18.288Z

Hi @chrscheuer,

Is there a way to combine the the scripts below to simulate a mouse drag from a UI elements relative position to an predefined x,y position?

sf.mouse.simulateDrag({
    startPosition: {"x":290,"y":10},
    endPosition: {"x":290,"y":28},
});
sf.ui.proTools.mainWindow.groups.allItems[7].mouseClickElement({
    relativePosition: {"x":290,"y":10},
    anchor: "TopLeft",
});
Solved in post #4, click to view
  • 25 replies

There are 25 replies. Estimated reading time: 25 minutes

  1. Hi Kitch!
    Yes instead of using mouseClickElement, access the element's frame property, from which you can get the screen coordinates to use as input for the simulateDrag event.
    However, depending on what you're trying to do it looks like there might be a more accurate solution. Right now you're fetching groups in the mainWindow by number for example - that won't be stable.
    Can you describe in more detail what you're trying to do?

    1. Kitch Membery @Kitch2020-04-18 22:48:21.676Z

      Hi mate... I hope you took yesterday off at least!?

      In it's simplest form I'm trying to drag track "Insert A" to track "Insert B". (As a part of a larger script to shuffle around inserts.) I'll eventually adapt this to a reusable function (and possible surface).

      Rock on!

      1. In reply tochrscheuer:
        Kitch Membery @Kitch2020-04-18 23:18:01.848Z2020-04-19 11:18:47.690Z

        Using the elements frame property works great... is there a way to make it work faster?

        var trackFrame = sf.ui.proTools.mainWindow.groups.allItems[7].frame;
        
        sf.mouse.simulateDrag({
            startPosition: {"x":trackFrame.x+290,"y":trackFrame.y+10},
            endPosition: {"x":trackFrame.x+290,"y":trackFrame.y+28},
        });
        
        ReplySolution
        1. Kitch Membery @Kitch2020-04-18 23:19:34.463Z2020-04-18 23:26:10.278Z

          Actually I think this is a Pro Tools limitation as I just tried it manually and noticed how slow it is by default depending on the plugin.

          1. There are many ways to optimize this :)

            Instead of using the simulateDrag you can manually perform the simulation by using the mouse.down mouse.drag and mouse.up methods in sequence.

            But more importantly, don't use relative positioning within the track at all. This will be highly unstable.
            Instead, get the frames from the buttons themselves:

            var insert1BtnFrame = sf.ui.proTools.selectedTrack.insertButtons[0].frame;
            var insert2BtnFrame = sf.ui.proTools.selectedTrack.insertButtons[1].frame;
            //...
            
            
            1. Kitch Membery @Kitch2020-04-20 09:08:21.619Z

              Great... Thanks mate! I'll make some changes in the script in my other post..

              1. For making it work faster, the crucial element is to figure out the timing of the events so that PT registers it.
                The simulateDrag function is designed to be slow enough to simulate how a human would move the mouse (and likely err on the slow side).

                If you use mouse.down, mouse.drag and mouse.up, the crucial point is the drag. As a minimum you need to fire off these events:

                • mouse.down at the start position
                • (Optional:) mouse.drag at points along the route
                • mouse.drag at the end position
                • sf.wait to make sure PT has registered you're hovering over a valid drop target
                • mouse.up at the end position

                Particularly the mouse.drag at the end position seems important. You might be able to get away with just a single mouse.drag call at the end position, and then a sf.wait so PT registers that the mouse is being dragged over a valid drop target, and then fire the mouse.up event.

                You can experiment with your own versions of this until you find some timing that's more optimized for this scenario.

                1. Kitch Membery @Kitch2020-04-20 11:07:48.938Z

                  That makes perfect sence and is a good lesson on speeding up scripts.

                  I think I need a little extra help with this though. I've never used the mouse.down, mouse.drag, mouse.up events before...

                  How would I do a simple drag insert1BtnFrame to insert2BtnFrame.

                  This is what I have so far (BTW is not optimised or working for that matter. :-)

                  //Activate Pro Tools
                  sf.ui.proTools.appActivateMainWindow();
                  
                  var insert1BtnFrame = sf.ui.proTools.selectedTrack.insertButtons[0].frame;
                  var insert2BtnFrame = sf.ui.proTools.selectedTrack.insertButtons[1].frame;
                  
                  sf.mouse.down({ position: { x: insert1BtnFrame.x+4, y: insert1BtnFrame.y+35 } });
                  sf.mouse.drag({ position: { x: insert2BtnFrame.x+4, y: insert2BtnFrame.y+35 } });
                  sf.wait({intervalMs:1000});
                  sf.mouse.up({ position: { x: insert2BtnFrame.x+4, y: insert2BtnFrame.y+35 } });
                  
                  1. Instead of writing this directly, write it as a function that essentially takes the same arguments as simulateDrag.

                    Something like this is what I would have in mind:

                    /**
                     * @param {PointF} startPos
                     * @param {PointF} endPos
                     */
                    function fastDrag(startPos, endPos)
                    {
                        const numberOfPointsInBetween = 3;
                        const timeToWaitBetweenEachDrag = 100;
                        const timeToWaitBeforeDropping = 500;
                    
                        sf.mouse.down({ position: startPos });
                    
                        for(var i=0; i<numberOfPointsInBetween; i++) {
                            var dragPos = {
                                x: (endPos.x - startPos.x) / numberOfPointsInBetween * i + startPos.x,
                                y: (endPos.y - startPos.y) / numberOfPointsInBetween * i + startPos.y,
                            };
                            sf.wait({ intervalMs: timeToWaitBetweenEachDrag });
                            sf.mouse.drag({ position: dragPos });
                        }
                    
                        sf.mouse.drag({ position: endPos });
                    
                        sf.wait({ intervalMs: timeToWaitBeforeDropping });
                        
                        sf.mouse.up({ position: endPos });
                    }
                    
                    1. Kitch Membery @Kitch2020-04-20 12:09:14.306Z

                      Sensational!
                      And It's 5am... oops!

                      1. Haha :)

                      2. In reply tochrscheuer:
                        Kitch Membery @Kitch2020-04-21 01:39:21.703Z2020-04-21 21:36:40.380Z

                        I'm getting so close with this one @chrscheuer,

                        However, intermittently the mouse.down event sometimes toggles the insert open/closed rather than moving it. I'm not sure why though. Any thoughts?

                        insertSlotSwitcher('Insert A', 'Insert B');
                        
                        function insertSlotSwitcher(originSlot, destinationSlot) {
                        
                            var insertButtonArray = ['Insert A', 'Insert B', 'Insert C', 'Insert D', 'Insert E', 'Insert F', 'Insert G', 'Insert H', 'Insert I', 'Insert J'];
                        
                            function fastDrag(startPos, endPos) {
                                const numberOfPointsInBetween = 3;
                                const timeToWaitBetweenEachDrag = 100;
                                const timeToWaitBeforeDropping = 1;
                        
                                sf.mouse.down({ position: startPos });
                        
                                for (var i = 0; i < numberOfPointsInBetween; i++) {
                                    var dragPos = {
                                        x: (endPos.x - startPos.x) / numberOfPointsInBetween * i + startPos.x,
                                        y: (endPos.y - startPos.y) / numberOfPointsInBetween * i + startPos.y,
                                    };
                                    sf.wait({ intervalMs: timeToWaitBetweenEachDrag });
                                    sf.mouse.drag({ position: dragPos });
                                }
                        
                                sf.mouse.drag({ position: endPos });
                        
                                sf.wait({ intervalMs: timeToWaitBeforeDropping });
                        
                                sf.mouse.up({ position: endPos });
                            }
                        
                            function getFirstFreeInsertIndex() {
                                var btns = sf.ui.proTools.selectedTrack.invalidate().insertButtons;
                                for (var i = 0; i < 10; i++)
                                    if (btns[i].value.invalidate().value === "unassigned") return i;
                                log("Script failed - There are no free insert slots");
                                throw 0;
                            }
                        
                            //Establish Index & Frame for First Free Slot
                            var firstFreeSlotIndex = getFirstFreeInsertIndex();
                            var firstFreeSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[firstFreeSlotIndex].frame;
                        
                            //Establish Index & Frame for Insert Origin
                            var originSlotIndex = insertButtonArray.indexOf(originSlot);
                            var originSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[originSlotIndex].frame;
                        
                            //Establish Index & Frame for  Insert Destination
                            var destinationSlotIndex = insertButtonArray.indexOf(destinationSlot);
                            var destinationSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].frame;
                        
                            //If originSlot is unassigned cancel script
                            if (sf.ui.proTools.selectedTrack.insertButtons[originSlotIndex].value.value === "unassigned") {
                                log("There is no insert on \"" + originSlot + "\"");
                                throw (0);
                            }
                        
                            //Perform Insert Moves 
                            if (sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].value.value === "unassigned") {
                                //Move Origin Slot to Sestination Slot
                                fastDrag(originSlotFrame, destinationSlotFrame);
                                
                            } else {
                                //Move Destination Slot to First Free Slot
                                fastDrag(destinationSlotFrame, firstFreeSlotFrame);
                        
                                //Wait for for First Free Slot to be assigned
                                while (sf.ui.proTools.selectedTrack.insertButtons[firstFreeSlotIndex].value.invalidate().value === "unassigned") {
                                    sf.wait({intervalMs:1});
                                }
                        
                                //Move Origin Slot to Sestination Slot
                                fastDrag(originSlotFrame, destinationSlotFrame);
                        
                                //Wait for for Destination Slot to be assigned
                                while (sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].value.invalidate().value === "unassigned") {
                                    sf.wait({intervalMs:1});
                                }
                                
                                //Move First Free Slot to Origin Slot
                                fastDrag(firstFreeSlotFrame, originSlotFrame);
                            };
                        }
                        

                        Also... I'm not sure I understand what the "numberOfPointsInBetween" is doing.

                        Thanks in advance. :-)

                        1. I think the reason for that is that there elapses too long time after your initial mouse.down until Pro Tools registers a drag, so it becomes registered as a click.

                          Try changing fastDrag to this:

                          function fastDrag(startPos, endPos) {
                              const numberOfPointsInBetween = 3;
                              const timeToWaitBeforeFirstDrag = 1;
                              const timeToWaitBetweenEachDrag = 100;
                              const timeToWaitBeforeDropping = 1;
                          
                              sf.mouse.down({ position: startPos });
                          
                              for (var i = 0; i < numberOfPointsInBetween; i++) {
                                  var dragPos = {
                                      x: (endPos.x - startPos.x) / numberOfPointsInBetween * i + startPos.x,
                                      y: (endPos.y - startPos.y) / numberOfPointsInBetween * i + startPos.y,
                                  };
                                  sf.wait({ intervalMs: i == 0 ? timeToWaitBeforeFirstDrag : timeToWaitBetweenEachDrag });
                                  sf.mouse.drag({ position: dragPos });
                              }
                          
                              sf.mouse.drag({ position: endPos });
                          
                              sf.wait({ intervalMs: timeToWaitBeforeDropping });
                          
                              sf.mouse.up({ position: endPos });
                          }
                          
                          1. Kitch Membery @Kitch2020-04-21 22:16:56.677Z

                            Thanks for that update to the fastDrag function. :-)

                            I noticed a few coding errors in my previous code and fixed them... I'd omitted a few ";" at the end of the fastDrag moves.

                            It seems to be more stable now but I still need to fully test it.

                            1. You might also need something else. The initial drag might have to be at least a certain amount of pixels for it to be registered as a drag. Worth noting if you're still seeing issues.

                              1. Kitch Membery @Kitch2020-04-22 18:23:57.098Z

                                It seems to be working ok now. Will test it more and see how it goes.

                                Here is the updated script that works on all selected tracks :-)

                                sf.ui.proTools.mainWindow.invalidate()
                                
                                sf.ui.proTools.selectedTracks.trackHeaders.slice().map(track => {
                                    track.trackSelect();
                                    insertSlotSwitcher('Insert D', 'Insert J');
                                });
                                
                                function insertSlotSwitcher(originSlot, destinationSlot) {
                                
                                    var insertButtonArray = ['Insert A', 'Insert B', 'Insert C', 'Insert D', 'Insert E', 'Insert F', 'Insert G', 'Insert H', 'Insert I', 'Insert J'];
                                
                                    function fastDrag(startPos, endPos) {
                                        const numberOfPointsInBetween = 3;
                                        const timeToWaitBeforeFirstDrag = 1;
                                        const timeToWaitBetweenEachDrag = 100;
                                        const timeToWaitBeforeDropping = 1;
                                
                                        sf.mouse.down({ position: startPos });
                                
                                        for (var i = 0; i < numberOfPointsInBetween; i++) {
                                            var dragPos = {
                                                x: (endPos.x - startPos.x) / numberOfPointsInBetween * i + startPos.x,
                                                y: (endPos.y - startPos.y) / numberOfPointsInBetween * i + startPos.y,
                                            };
                                            sf.wait({ intervalMs: i == 0 ? timeToWaitBeforeFirstDrag : timeToWaitBetweenEachDrag });
                                            sf.mouse.drag({ position: dragPos });
                                        }
                                
                                        sf.mouse.drag({ position: endPos });
                                
                                        sf.wait({ intervalMs: timeToWaitBeforeDropping });
                                
                                        sf.mouse.up({ position: endPos });
                                    }
                                
                                    function getFirstFreeInsertIndex() {
                                        var btns = sf.ui.proTools.selectedTrack.invalidate().insertButtons;
                                        for (var i = 0; i < 10; i++)
                                            if (btns[i].value.invalidate().value === "unassigned") return i;
                                    }
                                
                                    function checkForFreeInsert() {
                                        if (getFirstFreeInsertIndex() == undefined)
                                            return false
                                        else return true
                                    }
                                
                                
                                    if (checkForFreeInsert()) {
                                        //Establish Index & Frame for First Free Slot
                                        var firstFreeSlotIndex = getFirstFreeInsertIndex();
                                        var firstFreeSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[firstFreeSlotIndex].frame;
                                
                                        //Establish Index & Frame for Insert Origin
                                        var originSlotIndex = insertButtonArray.indexOf(originSlot);
                                        var originSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[originSlotIndex].frame;
                                
                                        //Establish Index & Frame for  Insert Destination
                                        var destinationSlotIndex = insertButtonArray.indexOf(destinationSlot);
                                        var destinationSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].frame;
                                    }
                                    //Perform Insert Moves 
                                
                                    if (checkForFreeInsert() === false) { //If no free slots are available
                                        log("There are no free insert slots on this track");
                                    } else if (sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].value.value === "unassigned") { //If Destination slot is unassigned
                                        //Move Origin Slot to Sestination Slot
                                        fastDrag(originSlotFrame, destinationSlotFrame);
                                    } else if (sf.ui.proTools.selectedTrack.insertButtons[originSlotIndex].value.value === "unassigned") { //If there is no Insert at the origin
                                        log("There is no insert on \"" + originSlot + "\"");
                                    } else {
                                        //Move Destination Slot to First Free Slot
                                        fastDrag(destinationSlotFrame, firstFreeSlotFrame);
                                
                                        //Wait for for First Free Slot to be assigned
                                        while (sf.ui.proTools.selectedTrack.insertButtons[firstFreeSlotIndex].value.invalidate().value === "unassigned") {
                                            sf.wait({ intervalMs: 1 });
                                        }
                                
                                        //Move Origin Slot to Sestination Slot
                                        fastDrag(originSlotFrame, destinationSlotFrame);
                                
                                        //Wait for for Destination Slot to be assigned
                                        while (sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].value.invalidate().value === "unassigned") {
                                            sf.wait({ intervalMs: 1 });
                                        }
                                
                                        //Move First Free Slot to Origin Slot
                                        fastDrag(firstFreeSlotFrame, originSlotFrame);
                                    };
                                }
                                
                                1. That's great!
                                  Noticing a slight issue here. You are using the .frame properties which are rectangles (ie. they both have topleft origin point as well as the width+height) and you give these to the fastDrag function to use, which expects to receive points. This works now because Rects have x and y properties, and points do too. But it's a little hacky.
                                  Also, it means you're using the very, very top left corner of each component's frame to do the drag, which may or may not be accurate. Normally you'd offset a couple of pixels on both x and y axis.

                                  Since it still makes sense to leave the fastDrag function to receive points (what would it need a rect for), you should convert the frames into points before you pass them into the fastDrag function.

                                  For example define a function getClickPoint that takes a frame as a parameter and returns a point with the x and y values incremented by 3.

                                  function getClickPoint(frame) {
                                      return {
                                          x: frame.x + 3,
                                          y: frame.y + 3,
                                      };
                                  }
                                  

                                  And then when you call fastDrag:

                                  fastDrag(getClickPoint(destinationSlotFrame), getClickPoint(firstFreeSlotFrame));
                                  
                                  1. Kitch Membery @Kitch2020-04-22 19:55:51.217Z

                                    Ahhh yes... Thanks Christian!

                                    Somthing like this?

                                    var insertFrame = sf.ui.proTools.selectedTrack.invalidate().insertButtons[1].frame;
                                    
                                    log(insertFrame);
                                    
                                    function getClickPoint(frame) {
                                        frame.x = frame.x + 3;
                                        frame.y = frame.y + 3;
                                        return frame
                                    }
                                    
                                    log(getClickPoint(insertFrame));
                                    
                                    1. Not exactly - you're still returning a frame instead of returning a point. A frame has 4 properties - x, y, w, h - a point has 2 - x and y.
                                      See my code above :)

                                      1. Kitch Membery @Kitch2020-04-22 20:15:35.772Z

                                        I see... I did it slightly different but I think it still works the same... (maybe?)

                                        I called the getClickPoint on the startPos and endPos within the fastdrag function.

                                        like this;

                                        function getClickPoint(frame) {
                                            frame.x = frame.x + 3;
                                            frame.y = frame.y + 3;
                                            return frame
                                        }
                                        
                                        function insertSlotSwitcher(originSlot, destinationSlot) {
                                        
                                            var insertButtonArray = ['Insert A', 'Insert B', 'Insert C', 'Insert D', 'Insert E', 'Insert F', 'Insert G', 'Insert H', 'Insert I', 'Insert J'];
                                        
                                            function fastDrag(startPos, endPos) {
                                        
                                                getClickPoint(startPos);
                                                getClickPoint(endPos);
                                        
                                                const numberOfPointsInBetween = 3;
                                                const timeToWaitBeforeFirstDrag = 1;
                                                const timeToWaitBetweenEachDrag = 100;
                                                const timeToWaitBeforeDropping = 1;
                                        
                                                sf.mouse.down({ position: startPos });
                                        
                                                for (var i = 0; i < numberOfPointsInBetween; i++) {
                                                    var dragPos = {
                                                        x: (endPos.x - startPos.x) / numberOfPointsInBetween * i + startPos.x,
                                                        y: (endPos.y - startPos.y) / numberOfPointsInBetween * i + startPos.y,
                                                    };
                                                    sf.wait({ intervalMs: i == 0 ? timeToWaitBeforeFirstDrag : timeToWaitBetweenEachDrag });
                                                    sf.mouse.drag({ position: dragPos });
                                                }
                                        
                                                sf.mouse.drag({ position: endPos });
                                        
                                                sf.wait({ intervalMs: timeToWaitBeforeDropping });
                                        
                                                sf.mouse.up({ position: endPos });
                                            }
                                        
                                        1. This is still wrong:

                                          function getClickPoint(frame) {
                                              frame.x = frame.x + 3;
                                              frame.y = frame.y + 3;
                                              return frame
                                          }
                                          

                                          See my posts above, I am saying the same thing over and over... You should not pass a frame to a function expecting a point. This could break in future versions of SoundFlow.
                                          You are returning a frame. Don't return a frame.

                                          1. You are also modifying an object instead of creating a new one. This is not ideal either.
                                            Now the function is called getClickPoint but it doesn't get a point, it modifies a frame.
                                            So this is going in the wrong direction...

                                            1. Now that I read your post again, this is even worse, because you moved the logic into fastDrag. That definitely doesn't make sense.
                                              What I was trying to improve in your code was to keep the fastDrag function only worrying about dragging between two points - keep it that way. You just need to feed it the two correct points, calculated from the frames. Hence my initial post.

                                              1. Kitch Membery @Kitch2020-04-22 20:51:55.219Z

                                                Thanks. I get it now.

                                                Sorry if I frustrated you.

                                                1. All good. Long day haha, and been dealing with Stream Decks, which is always pretty awful... :)