Hi @chrscheuer,
Is there a way to combine the the scripts below to simulate a mouse drag from a UI elements relative position to an predefined x,y position?
sf.mouse.simulateDrag({
startPosition: {"x":290,"y":10},
endPosition: {"x":290,"y":28},
});
sf.ui.proTools.mainWindow.groups.allItems[7].mouseClickElement({
relativePosition: {"x":290,"y":10},
anchor: "TopLeft",
});
Linked from:
- Christian Scheuer @chrscheuer2020-04-18 22:39:25.714Z
Hi Kitch!
Yes instead of using mouseClickElement, access the element's frame property, from which you can get the screen coordinates to use as input for the simulateDrag event.
However, depending on what you're trying to do it looks like there might be a more accurate solution. Right now you're fetching groups in the mainWindow by number for example - that won't be stable.
Can you describe in more detail what you're trying to do?Kitch Membery @Kitch2020-04-18 22:48:21.676Z
Hi mate... I hope you took yesterday off at least!?
In it's simplest form I'm trying to drag track "Insert A" to track "Insert B". (As a part of a larger script to shuffle around inserts.) I'll eventually adapt this to a reusable function (and possible surface).
Rock on!
- In reply tochrscheuer⬆:
Kitch Membery @Kitch2020-04-18 23:18:01.848Z2020-04-19 11:18:47.690Z
Using the elements frame property works great... is there a way to make it work faster?
var trackFrame = sf.ui.proTools.mainWindow.groups.allItems[7].frame; sf.mouse.simulateDrag({ startPosition: {"x":trackFrame.x+290,"y":trackFrame.y+10}, endPosition: {"x":trackFrame.x+290,"y":trackFrame.y+28}, });
Kitch Membery @Kitch2020-04-18 23:19:34.463Z2020-04-18 23:26:10.278Z
Actually I think this is a Pro Tools limitation as I just tried it manually and noticed how slow it is by default depending on the plugin.
Christian Scheuer @chrscheuer2020-04-20 08:39:54.855Z
There are many ways to optimize this :)
Instead of using the
simulateDrag
you can manually perform the simulation by using themouse.down
mouse.drag
andmouse.up
methods in sequence.But more importantly, don't use relative positioning within the track at all. This will be highly unstable.
Instead, get the frames from the buttons themselves:var insert1BtnFrame = sf.ui.proTools.selectedTrack.insertButtons[0].frame; var insert2BtnFrame = sf.ui.proTools.selectedTrack.insertButtons[1].frame; //...
Kitch Membery @Kitch2020-04-20 09:08:21.619Z
Great... Thanks mate! I'll make some changes in the script in my other post..
Christian Scheuer @chrscheuer2020-04-20 10:26:27.513Z
For making it work faster, the crucial element is to figure out the timing of the events so that PT registers it.
ThesimulateDrag
function is designed to be slow enough to simulate how a human would move the mouse (and likely err on the slow side).If you use mouse.down, mouse.drag and mouse.up, the crucial point is the drag. As a minimum you need to fire off these events:
- mouse.down at the start position
- (Optional:) mouse.drag at points along the route
- mouse.drag at the end position
- sf.wait to make sure PT has registered you're hovering over a valid drop target
- mouse.up at the end position
Particularly the mouse.drag at the end position seems important. You might be able to get away with just a single mouse.drag call at the end position, and then a sf.wait so PT registers that the mouse is being dragged over a valid drop target, and then fire the mouse.up event.
You can experiment with your own versions of this until you find some timing that's more optimized for this scenario.
Kitch Membery @Kitch2020-04-20 11:07:48.938Z
That makes perfect sence and is a good lesson on speeding up scripts.
I think I need a little extra help with this though. I've never used the mouse.down, mouse.drag, mouse.up events before...
How would I do a simple drag insert1BtnFrame to insert2BtnFrame.
This is what I have so far (BTW is not optimised or working for that matter. :-)
//Activate Pro Tools sf.ui.proTools.appActivateMainWindow(); var insert1BtnFrame = sf.ui.proTools.selectedTrack.insertButtons[0].frame; var insert2BtnFrame = sf.ui.proTools.selectedTrack.insertButtons[1].frame; sf.mouse.down({ position: { x: insert1BtnFrame.x+4, y: insert1BtnFrame.y+35 } }); sf.mouse.drag({ position: { x: insert2BtnFrame.x+4, y: insert2BtnFrame.y+35 } }); sf.wait({intervalMs:1000}); sf.mouse.up({ position: { x: insert2BtnFrame.x+4, y: insert2BtnFrame.y+35 } });
Christian Scheuer @chrscheuer2020-04-20 11:47:52.102Z
Instead of writing this directly, write it as a function that essentially takes the same arguments as simulateDrag.
Something like this is what I would have in mind:
/** * @param {PointF} startPos * @param {PointF} endPos */ function fastDrag(startPos, endPos) { const numberOfPointsInBetween = 3; const timeToWaitBetweenEachDrag = 100; const timeToWaitBeforeDropping = 500; sf.mouse.down({ position: startPos }); for(var i=0; i<numberOfPointsInBetween; i++) { var dragPos = { x: (endPos.x - startPos.x) / numberOfPointsInBetween * i + startPos.x, y: (endPos.y - startPos.y) / numberOfPointsInBetween * i + startPos.y, }; sf.wait({ intervalMs: timeToWaitBetweenEachDrag }); sf.mouse.drag({ position: dragPos }); } sf.mouse.drag({ position: endPos }); sf.wait({ intervalMs: timeToWaitBeforeDropping }); sf.mouse.up({ position: endPos }); }
Kitch Membery @Kitch2020-04-20 12:09:14.306Z
Sensational!
And It's 5am... oops!Christian Scheuer @chrscheuer2020-04-20 15:33:11.622Z
Haha :)
- In reply tochrscheuer⬆:
Kitch Membery @Kitch2020-04-21 01:39:21.703Z2020-04-21 21:36:40.380Z
I'm getting so close with this one @chrscheuer,
However, intermittently the mouse.down event sometimes toggles the insert open/closed rather than moving it. I'm not sure why though. Any thoughts?
insertSlotSwitcher('Insert A', 'Insert B'); function insertSlotSwitcher(originSlot, destinationSlot) { var insertButtonArray = ['Insert A', 'Insert B', 'Insert C', 'Insert D', 'Insert E', 'Insert F', 'Insert G', 'Insert H', 'Insert I', 'Insert J']; function fastDrag(startPos, endPos) { const numberOfPointsInBetween = 3; const timeToWaitBetweenEachDrag = 100; const timeToWaitBeforeDropping = 1; sf.mouse.down({ position: startPos }); for (var i = 0; i < numberOfPointsInBetween; i++) { var dragPos = { x: (endPos.x - startPos.x) / numberOfPointsInBetween * i + startPos.x, y: (endPos.y - startPos.y) / numberOfPointsInBetween * i + startPos.y, }; sf.wait({ intervalMs: timeToWaitBetweenEachDrag }); sf.mouse.drag({ position: dragPos }); } sf.mouse.drag({ position: endPos }); sf.wait({ intervalMs: timeToWaitBeforeDropping }); sf.mouse.up({ position: endPos }); } function getFirstFreeInsertIndex() { var btns = sf.ui.proTools.selectedTrack.invalidate().insertButtons; for (var i = 0; i < 10; i++) if (btns[i].value.invalidate().value === "unassigned") return i; log("Script failed - There are no free insert slots"); throw 0; } //Establish Index & Frame for First Free Slot var firstFreeSlotIndex = getFirstFreeInsertIndex(); var firstFreeSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[firstFreeSlotIndex].frame; //Establish Index & Frame for Insert Origin var originSlotIndex = insertButtonArray.indexOf(originSlot); var originSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[originSlotIndex].frame; //Establish Index & Frame for Insert Destination var destinationSlotIndex = insertButtonArray.indexOf(destinationSlot); var destinationSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].frame; //If originSlot is unassigned cancel script if (sf.ui.proTools.selectedTrack.insertButtons[originSlotIndex].value.value === "unassigned") { log("There is no insert on \"" + originSlot + "\""); throw (0); } //Perform Insert Moves if (sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].value.value === "unassigned") { //Move Origin Slot to Sestination Slot fastDrag(originSlotFrame, destinationSlotFrame); } else { //Move Destination Slot to First Free Slot fastDrag(destinationSlotFrame, firstFreeSlotFrame); //Wait for for First Free Slot to be assigned while (sf.ui.proTools.selectedTrack.insertButtons[firstFreeSlotIndex].value.invalidate().value === "unassigned") { sf.wait({intervalMs:1}); } //Move Origin Slot to Sestination Slot fastDrag(originSlotFrame, destinationSlotFrame); //Wait for for Destination Slot to be assigned while (sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].value.invalidate().value === "unassigned") { sf.wait({intervalMs:1}); } //Move First Free Slot to Origin Slot fastDrag(firstFreeSlotFrame, originSlotFrame); }; }
Also... I'm not sure I understand what the "numberOfPointsInBetween" is doing.
Thanks in advance. :-)
Christian Scheuer @chrscheuer2020-04-21 11:27:17.642Z
I think the reason for that is that there elapses too long time after your initial mouse.down until Pro Tools registers a drag, so it becomes registered as a click.
Try changing
fastDrag
to this:function fastDrag(startPos, endPos) { const numberOfPointsInBetween = 3; const timeToWaitBeforeFirstDrag = 1; const timeToWaitBetweenEachDrag = 100; const timeToWaitBeforeDropping = 1; sf.mouse.down({ position: startPos }); for (var i = 0; i < numberOfPointsInBetween; i++) { var dragPos = { x: (endPos.x - startPos.x) / numberOfPointsInBetween * i + startPos.x, y: (endPos.y - startPos.y) / numberOfPointsInBetween * i + startPos.y, }; sf.wait({ intervalMs: i == 0 ? timeToWaitBeforeFirstDrag : timeToWaitBetweenEachDrag }); sf.mouse.drag({ position: dragPos }); } sf.mouse.drag({ position: endPos }); sf.wait({ intervalMs: timeToWaitBeforeDropping }); sf.mouse.up({ position: endPos }); }
Kitch Membery @Kitch2020-04-21 22:16:56.677Z
Thanks for that update to the fastDrag function. :-)
I noticed a few coding errors in my previous code and fixed them... I'd omitted a few ";" at the end of the fastDrag moves.
It seems to be more stable now but I still need to fully test it.
Christian Scheuer @chrscheuer2020-04-22 11:08:30.902Z
You might also need something else. The initial drag might have to be at least a certain amount of pixels for it to be registered as a drag. Worth noting if you're still seeing issues.
Kitch Membery @Kitch2020-04-22 18:23:57.098Z
It seems to be working ok now. Will test it more and see how it goes.
Here is the updated script that works on all selected tracks :-)
sf.ui.proTools.mainWindow.invalidate() sf.ui.proTools.selectedTracks.trackHeaders.slice().map(track => { track.trackSelect(); insertSlotSwitcher('Insert D', 'Insert J'); }); function insertSlotSwitcher(originSlot, destinationSlot) { var insertButtonArray = ['Insert A', 'Insert B', 'Insert C', 'Insert D', 'Insert E', 'Insert F', 'Insert G', 'Insert H', 'Insert I', 'Insert J']; function fastDrag(startPos, endPos) { const numberOfPointsInBetween = 3; const timeToWaitBeforeFirstDrag = 1; const timeToWaitBetweenEachDrag = 100; const timeToWaitBeforeDropping = 1; sf.mouse.down({ position: startPos }); for (var i = 0; i < numberOfPointsInBetween; i++) { var dragPos = { x: (endPos.x - startPos.x) / numberOfPointsInBetween * i + startPos.x, y: (endPos.y - startPos.y) / numberOfPointsInBetween * i + startPos.y, }; sf.wait({ intervalMs: i == 0 ? timeToWaitBeforeFirstDrag : timeToWaitBetweenEachDrag }); sf.mouse.drag({ position: dragPos }); } sf.mouse.drag({ position: endPos }); sf.wait({ intervalMs: timeToWaitBeforeDropping }); sf.mouse.up({ position: endPos }); } function getFirstFreeInsertIndex() { var btns = sf.ui.proTools.selectedTrack.invalidate().insertButtons; for (var i = 0; i < 10; i++) if (btns[i].value.invalidate().value === "unassigned") return i; } function checkForFreeInsert() { if (getFirstFreeInsertIndex() == undefined) return false else return true } if (checkForFreeInsert()) { //Establish Index & Frame for First Free Slot var firstFreeSlotIndex = getFirstFreeInsertIndex(); var firstFreeSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[firstFreeSlotIndex].frame; //Establish Index & Frame for Insert Origin var originSlotIndex = insertButtonArray.indexOf(originSlot); var originSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[originSlotIndex].frame; //Establish Index & Frame for Insert Destination var destinationSlotIndex = insertButtonArray.indexOf(destinationSlot); var destinationSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].frame; } //Perform Insert Moves if (checkForFreeInsert() === false) { //If no free slots are available log("There are no free insert slots on this track"); } else if (sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].value.value === "unassigned") { //If Destination slot is unassigned //Move Origin Slot to Sestination Slot fastDrag(originSlotFrame, destinationSlotFrame); } else if (sf.ui.proTools.selectedTrack.insertButtons[originSlotIndex].value.value === "unassigned") { //If there is no Insert at the origin log("There is no insert on \"" + originSlot + "\""); } else { //Move Destination Slot to First Free Slot fastDrag(destinationSlotFrame, firstFreeSlotFrame); //Wait for for First Free Slot to be assigned while (sf.ui.proTools.selectedTrack.insertButtons[firstFreeSlotIndex].value.invalidate().value === "unassigned") { sf.wait({ intervalMs: 1 }); } //Move Origin Slot to Sestination Slot fastDrag(originSlotFrame, destinationSlotFrame); //Wait for for Destination Slot to be assigned while (sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].value.invalidate().value === "unassigned") { sf.wait({ intervalMs: 1 }); } //Move First Free Slot to Origin Slot fastDrag(firstFreeSlotFrame, originSlotFrame); }; }
Christian Scheuer @chrscheuer2020-04-22 18:55:52.944Z
That's great!
Noticing a slight issue here. You are using the.frame
properties which are rectangles (ie. they both have topleft origin point as well as the width+height) and you give these to the fastDrag function to use, which expects to receive points. This works now because Rects have x and y properties, and points do too. But it's a little hacky.
Also, it means you're using the very, very top left corner of each component's frame to do the drag, which may or may not be accurate. Normally you'd offset a couple of pixels on both x and y axis.Since it still makes sense to leave the fastDrag function to receive points (what would it need a rect for), you should convert the frames into points before you pass them into the fastDrag function.
For example define a function
getClickPoint
that takes a frame as a parameter and returns a point with the x and y values incremented by 3.function getClickPoint(frame) { return { x: frame.x + 3, y: frame.y + 3, }; }
And then when you call fastDrag:
fastDrag(getClickPoint(destinationSlotFrame), getClickPoint(firstFreeSlotFrame));
Kitch Membery @Kitch2020-04-22 19:55:51.217Z
Ahhh yes... Thanks Christian!
Somthing like this?
var insertFrame = sf.ui.proTools.selectedTrack.invalidate().insertButtons[1].frame; log(insertFrame); function getClickPoint(frame) { frame.x = frame.x + 3; frame.y = frame.y + 3; return frame } log(getClickPoint(insertFrame));
Christian Scheuer @chrscheuer2020-04-22 20:07:05.920Z
Not exactly - you're still returning a frame instead of returning a point. A frame has 4 properties - x, y, w, h - a point has 2 - x and y.
See my code above :)Kitch Membery @Kitch2020-04-22 20:15:35.772Z
I see... I did it slightly different but I think it still works the same... (maybe?)
I called the getClickPoint on the startPos and endPos within the fastdrag function.
like this;
function getClickPoint(frame) { frame.x = frame.x + 3; frame.y = frame.y + 3; return frame } function insertSlotSwitcher(originSlot, destinationSlot) { var insertButtonArray = ['Insert A', 'Insert B', 'Insert C', 'Insert D', 'Insert E', 'Insert F', 'Insert G', 'Insert H', 'Insert I', 'Insert J']; function fastDrag(startPos, endPos) { getClickPoint(startPos); getClickPoint(endPos); const numberOfPointsInBetween = 3; const timeToWaitBeforeFirstDrag = 1; const timeToWaitBetweenEachDrag = 100; const timeToWaitBeforeDropping = 1; sf.mouse.down({ position: startPos }); for (var i = 0; i < numberOfPointsInBetween; i++) { var dragPos = { x: (endPos.x - startPos.x) / numberOfPointsInBetween * i + startPos.x, y: (endPos.y - startPos.y) / numberOfPointsInBetween * i + startPos.y, }; sf.wait({ intervalMs: i == 0 ? timeToWaitBeforeFirstDrag : timeToWaitBetweenEachDrag }); sf.mouse.drag({ position: dragPos }); } sf.mouse.drag({ position: endPos }); sf.wait({ intervalMs: timeToWaitBeforeDropping }); sf.mouse.up({ position: endPos }); }
Christian Scheuer @chrscheuer2020-04-22 20:18:29.769Z
This is still wrong:
function getClickPoint(frame) { frame.x = frame.x + 3; frame.y = frame.y + 3; return frame }
See my posts above, I am saying the same thing over and over... You should not pass a frame to a function expecting a point. This could break in future versions of SoundFlow.
You are returning a frame. Don't return a frame.Christian Scheuer @chrscheuer2020-04-22 20:19:31.260Z
You are also modifying an object instead of creating a new one. This is not ideal either.
Now the function is called getClickPoint but it doesn't get a point, it modifies a frame.
So this is going in the wrong direction...Christian Scheuer @chrscheuer2020-04-22 20:21:05.999Z
Now that I read your post again, this is even worse, because you moved the logic into fastDrag. That definitely doesn't make sense.
What I was trying to improve in your code was to keep the fastDrag function only worrying about dragging between two points - keep it that way. You just need to feed it the two correct points, calculated from the frames. Hence my initial post.Kitch Membery @Kitch2020-04-22 20:51:55.219Z
Thanks. I get it now.
Sorry if I frustrated you.
Christian Scheuer @chrscheuer2020-04-22 21:01:20.572Z
All good. Long day haha, and been dealing with Stream Decks, which is always pretty awful... :)